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ABSTRACT 

The  Janus  wargame  was  assessed  as  a  means  of  investigating  naturalistic  decision-making 
(NDM).  A  further  aim  was  to  establish  the  generality  of  previous  research  that  uses  non-military 
simulation.  Participants  were  divided  into  hierarchically  structured  teams  of  3  (one  military  team, 
and  three  civilian  teams).  Each  team  was  tested  using  open  and  restricted  communication 
architectures.  In  line  with  predictions,  open  communication  was  more  effective  than  restricted 
communication.  In  addition,  military  personnel  out-performed  civilian  participants.  No  linear  or 
quadratic  patterns  were  found  regarding  the  development  of  expertise.  It  was  concluded  that 
Janus  was  an  effective  means  of  examining  NDM.  In  addition,  the  data  indicated  that  non¬ 
military  simulation  can  generate  valid  data  in  relation  to  communication  architectures,  but  not  in 
relation  to  the  development  of  military  expertise. 
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The  Use  of  the  Janus  War  game  Simulation  to 
Investigate  Naturalistic  Decision-Making: 
A  Preliminary  Examination 


Executive  Summary 

Over  recent  years  there  has  been  an  increase  in  the  introduction  of  Information 
Technology  (IT)  to  military  headquarters  (HQ),  To  achieve  effective  and  efficient 
functioning  in  the  Defence  Force  it  is  important  for  research  to  be  conducted  that  identifies 
the  following  factors:  a)  the  development  of  expertise;  and  b)  how  communication 
facilitates  situation  awareness.  It  is  also  important  to  identify  an  appropriate  context  for 
testing  these  processes.  Consequently  the  aim  of  this  research  was  to  assess  the  use  of  the 
Janus  wargame  simulation  as  a  means  of  investigating  the  impact  of  IT  on  naturalistic 
decision-making  (NDM).  There  was  also  an  examination  of  the  development  of  expertise 
and  the  role  of  communication  architecture  in  facilitating  situation  awareness.  A  further 
aim  was  to  establish  the  generality  of  results  from  research  that  uses  non-military 
microworlds  (eg.  Networked  Fire  Chief  (NFC)). 

Janus  models  realistic  conditions  and  presents  the  user  with  pressures  similar  to  those 
found  in  a  real  world  NDM  situation  (i.e.  time  pressure,  uncertainty,  and  high  stakes).  In 
the  current  study,  it  was  networked  with  a  situation  awareness  tool  to  simulate  a 
simplistic  command  and  control  (C2)  hierarchy.  Experiment  participants  were  divided 
into  hierarchically  structured  teams  of  3  (one  military  team,  and  three  civilian  teams).  One 
team  member  acted  as  the  commander,  while  the  other  two  played  the  ground  force 
controller  and  the  support  force  controller.  The  teams  were  instructed  to  work  together  to 
accomplish  the  goals  defined  in  an  intent  statement.  This  involved  an  initial  planning 
session,  followed  by  the  wargame,  where  participants  were  tested  under  the  following 
conditions: 

•  Open  communication:  All  team  members  hear  all  communication. 

•  Restricted  communication:  Communication  is  limited  to  the  commander  and  one  of  his 

subordinates  at  any  one  time.  No  one  else  in  the  team  can  hear  the  transmission. 

Participants  communicated  via  radios  to  coordinate  their  actions.  Testing  was  conducted 
over  three  days,  and  involved  8  Janus  trials,  each  lasting  a  maximum  of  V/i  hours. 

Janus  performance  scores  were  calculated  in  terms  of  the  loss  exchange  ratio  (number  of 
enemy  kills  divided  by  total  losses).  Team  interactions  were  also  monitored  to  provide  a 
measure  of  communication  problems,  and  behaviours  that  assist  in  the  maintenance  and 
development  of  situation  awareness.  Questionnaires  were  used  to  collect  demographic 
information  and  to  assess  the  participant's  perception  of  their  own  progress. 


In  summary,  the  data  indicated  that: 

•  Open  communication  was  more  effective  than  restricted  communication; 

•  The  majority  of  participants  preferred  using  open  communication; 

•  Military  personnel  outperformed  civilian  participants; 

•  Military  participants  used  substantially  more  acknowledgments  and  update  reports  than 

civilians,  communication  that  facilitates  teamwork  and  situation  awareness; 

•  The  development  of  expertise  across  trials  showed  no  clear  pattern; 

•  Participants  felt  their  situation  awareness  and  level  of  expertise  increased  over  the  trials; 

and 

•  Due  to  lack  of  expertise,  civilian  participants  did  not  know  which  information  to  filter 

out,  so  were  requesting  more  than  was  necessary  for  optimal  performance. 

In  summary,  Janus  was  an  effective  means  of  examining  IT  and  NDM.  In  addition,  the 
data  indicated  that  NFC  generates  valid  data  in  relation  to  communication  architecture 
and  military  performance.  However,  the  data  suggests  that  expertise  is  domain  specific,  so 
NFC  may  not  be  an  appropriate  method  of  examining  the  development  of  military 
expertise. 
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1.  Introduction 

Over  recent  years  there  has  been  a  vast  increase  in  the  introduction  of  Information 
Technology  (IT)  to  the  workplace.  For  example,  the  Australian  Army  has  been  introducing 
the  Battlefield  Command  Support  System  (BCSS)  to  the  various  Brigades  (Bde).  BCSS  is  a 
suite  of  software  tools  designed  to  assist  command  decision-making  in  the  Australian 
Army.  Despite  the  promise  offered  by  new  technology  difficulties  in  the  use  of  IT  systems 
have  emerged  (Mills,  2000;  Huf,  2001).  Some  of  these  difficulties  relate  to  providing  new 
technology  to  users  with  varying  levels  of  expertise.  In  addition,  IT  systems  have  the 
potential  to  enhance  or  interfere  with  team  decision-making  and  situation  awareness 
(Mills,  2000;  Huf,  2001).  To  achieve  effective  and  efficient  functioning  in  the  Defence  Force, 
it  is  therefore  important  for  research  to  be  conducted  that  identifies  the  following  factors: 
a)  the  development  of  expertise;  and  b)  how  communication  facilitates  situation 
awareness.  It  is  also  important  to  identify  an  appropriate  context  for  testing  these 
processes.  Consequently,  this  report  details  an  assessment  of  communication  architecture 
and  previous  experience  in  relation  to  decision-making  using  a  simulated  wargame.  It  also 
describes  the  results  of  an  investigation  of  the  development  of  expertise. 


2,  Naturalistic  Decision-Making 

Research  on  naturalistic  decision-making  (NDM)  is  a  relatively  recent  research  area  in 
psychology.  It  is  currently  being  applied  in  practical  situations,  such  as  interface  design 
(Hopple,  1988),  the  production  of  appropriate  decision-making  tools,  and  team  training 
(Zsambok  &  Klein,  1997).  NDM  theory  is  concerned  with  decision-making  in  dynamic, 
uncertain,  fast  paced  environments.  It  acts  as  an  alternative  explanation  to  classical 
decision  theory  for  decision-making  tasks  that  do  not  fit  "into  the  rather  limited  mould 
that  classical  decision  theory  provides"  (Beach  and  Lipshitz,  1993,  p.  23). 

2.1  The  Classical  Decision-Making  Model 

Edwards  (1954)  introduced  the  concept  of  classical  decision-making  to  the  field  of 
psychology.  The  model  is  used  to  investigate  everyday  decision-making  processes. 
Classical  theory  suggests  that  people  collect  and  analyse  information,  eventually  selecting 
an  optimal  solution  from  a  range  of  alternatives.  This  is  done  by  evaluating  the  advantages 
and  disadvantages  of  each  possible  outcome  and  choosing  the  one  most  appropriate  to 
achieve  the  desired  outcome  goal.  This  decision  is  regarded  as  optimal  (McDaniel,  1993). 

Classical  decision-making  research  investigates  the  quality  of  the  decision  by  comparing  it 
with  a  normative  statistical  model.  It  relies  on  laboratory  experiments  to  examine  how  a 
decision-making  outcome  differs  from  the  optimal  solution  (Brehmer,  Jungermann, 
Lourens  &  Sevon,  1985).  Thus,  rather  than  studying  how  people  actually  make  decisions, 
classical  decision  theory  focuses  on  how  the  optimal  decision  should  be  reached. 

A  potential  problem  with  this  approach  is  that  it  does  not  take  contextual  factors  into 
account.  Classical  decision  theory  has  been  found  to  encounter  problems  when  variables 
such  as  time  pressure,  uncertainty,  complexity  and  high  stakes  are  involved  in  the 
decision-making  process.  Needing  to  evaluate  such  a  large  number  of  alternatives  in  a 
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short  period  of  time  could  produce  cognitive  overload.  This  could  result  in  serious  errors. 
In  such  circumstances,  there  are  grounds  for  querying  the  adequacy  of  the  classical 
decision  model. 

2.1.1  Defining  Naturalistic  Decision  Making  (NDM) 

The  search  for  a  more  appropriate  model  of  decision-making  under  these  conditions  has 
seen  the  development  of  NDM  theory  (Cannon-Bowers,  Salas  &  Pruitt,  1996).  Zsambok 
(1997)  defines  NDM  as 

how  experienced  people,  working  as  individuals  or  groups  in  dynamic,  uncertain,  and 
often  fast  paced  environments,  identify  and  assess  their  situation,  make  decisions  and 
take  actions  whose  consequences  are  meaningful  to  them  and  to  the  larger  organisation 
in  which  they  operate  (p.  5). 

In  other  words,  NDM  research  investigates  how  people  use  experience  to  make  decisions 
in  naturalistic  environments  (eg.  under  time  pressure,  shifting  conditions,  unclear  goals, 
degraded  information  and  within  team  interactions).  Cannon-Bowers,  Salas  and  Pruitt 
(1996)  suggest  that  NDM  theory  represents  a  more  appropriate  model  for  decision-making 
in  naturalistic  environments,  as  it  takes  into  account  the  real  world  pressures.  Figure  1 
illustrates  the  circumstances  under  which  the  two  decision-making  models  are  suitable. 


: — u 


Figure  1.  Comparing  the  NDM  model  with  classical  decision-making  (Klein,  1997) 

2.1.2  Naturalistic  Decision-Making  Processes:  Klein's  RPDM  Model 

Based  on  research  on  decision-making  in  natural  environments,  Klein  (1989)  formed  the 
recognition  primed  decision-making  (RPDM)  model.  The  model  was  based  on  interviews 
and  observations  of  fire  ground  commanders  working  in  difficult  and  challenging 
circumstances  (Klein,  Calderwood  &  Clinton-Cirocco,  1986).  It  was  thought  that  under 
complex,  time-pressured  circumstances,  the  commanders  would  make  limited 
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comparisons  between  possible  outcomes.  However,  it  was  revealed  that  they  were  making 
no  comparisons  at  all.  In  fact,  80%  of  decisions  made  in  this  task  were  made  in  less  than 
one  minute.  From  this,  Klein  et  al.  (1986)  identified  the  following  issues: 

1-  The  commanders  drew  on  their  previous  experience  to  recognise  a  typical  action  to 
take; 

2.  They  did  not  have  to  find  an  optimal  solution,  merely  a  workable  one;  and 

3.  Once  they  arrived  at  a  suitable  course  of  action,  they  would  mentally  simulate  it  first  to 
ensure  it  would  work. 

Consequently,  Klein  (1989)  suggested  the  RPDM  model  of  3STDM.  The  RPDM  model 
implies  that  experienced  decision-makers: 

1.  Usually  consider  a  workable  option  and  do  not  need  to  generate  a  large  set  of 
alternatives; 

2.  Generate  and  evaluate  options  one  at  a  time,  instead  of  comparing  the  advantages  and 
disadvantages  of  all  options;  and 

3.  Evaluate  an  option  by  imagining  the  outcome,  and  by  finding  ways  to  avoid  problems 
that  may  arise  from  its  implementation. 

4.  Focus  on  assessing  the  situation  and  looking  for  familiar  cues. 

5.  Emphasise  acting  quickly  and  not  sustaining  analysis. 

The  RPDM  model  describes  how  people  make  decisions  without  comparing  outcomes  (see 
Figure  2).  The  decision-maker  initially  assesses  the  situation,  looking  for  familiar  patterns 
or  prototypes.  This  allows  him/her  to  know  which  goals  make  sense,  what  relevant  cues 
to  expect,  and  what  action  should  be  appropriate.  A  series  of  options  is  then  generated. 
The  first  solution  may  not  be  optimal,  but  it  will  usually  be  workable.  This  model  also 
shows  that  action  can  be  taken  quickly,  which  is  important  in  crisis  management  (Klein, 
1997).  The  RPDM  model  has  been  replicated  and  is  seen  as  a  viable  explanation  of  the 
cognitive  processes  underlying  NDM  (Kaemf,  Wolf,  Thordsen,  &  Klein,  1996;  Mosier, 
1990;  Pascual  &  Henderson,  in  press;  Randel,  Pugh,  Reed,  Schuler  &  Wyman,  1994). 
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Figure  2.  The  RPDM  model  (Klein,  2000) 

2.1.3  NDM  Themes 

According  to  Zsambok  (1997)  NDM  has  four  main  themes: 

1.  The  tasks  and  setting  involve  poorly  structured  problems,  uncertain  and  dynamic 
environments,  shifting  or  ill  defined  goals,  action  feedback  loops,  time  pressure,  high 
stakes,  teamwork,  and  organisational  goals  and  norms; 

2.  The  process  of  decision-making  relies  on  developing  situation  awareness,  diagnosis 
and  plan  generation  rather  than  focusing  on  a  moment  of  choice; 

3.  Decision-makers  draw  on  previous  experience  to  make  current  decisions;  and 

4.  The  aim  of  NDM  research  is  primarily  to  describe  more  accurately  and  understand  the 
decision-making  processes  people  use,  rather  than  investigating  how  people's  decisions 
differ  from  statistically  generated  optimal  solutions. 

Brehmer  (1997)  reinforced  Zsambok's  description  of  a  naturalistic  environment, 
suggesting  that  the  characteristics  include  complexity,  feedback  quality,  delay,  rate  of 
change,  how  the  environment  is  affected  by  the  decision,  and  the  extent  of  delegation  that 
is  used.  Some  examples  of  areas  in  which  NDM  research  has  been  applied  are  during 
military  operations,  fire  fighting,  in  operating  theatres,  nuclear  power  plants  and  oil  rigs. 
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2.1.4  Situation  Awareness 

Klein  and  Calderwood  (1991)  describe  NDM  as  continual  situation  assessment.  This 
contrasts  with  singular  moments  of  choice  described  in  the  classical  decision-making 
model.  NDM  often  requires  a  series  of  decisions  that  are  interdependent.  The  decision¬ 
maker  must  constantly  monitor  the  environment  that  will  be  spontaneously  changing,  as 
well  as  changing  because  of  each  decision.  Continual  assessment  produces  situation 
awareness.  The  initial  part  of  the  RPDM  model  outlines  recognition  of  a  situation.  The  rest 
of  the  decision-making  process  depends  on  correct  judgement  of  the  situation. 

George,  Kaempf,  Klein,  Thorsden  and  Wolf  (1996)  found  that  "awareness  of  the  situation 
enabled  officers  to  recognise  appropriate  actions  from  published  procedures,  and  past 
experience"  (p,220).  They  identified  the  following  methods  of  arriving  at  situation 
awareness: 

1.  87%  of  participants  used  a  feature  matching  strategy.  This  is  where  the  decision-maker 
sees  the  situation  as  familiar,  and  arrives  at  situation  awareness  through  a  series  of 
recognised  cues; 

2.  12%  of  participants  used  story  generation.  When  the  environment  does  not  provide 
enough  information  to  be  recognised  as  familiar,  the  decision-maker  constructs  a  story 
to  explain  the  information  and  to  arrive  at  greater  situation  awareness;  and 

3.  1%  of  cases  did  not  fit  into  either  of  these  main  categories. 

Developing  good  situation  awareness  is  an  important  part  of  NDM.  It  allows  decision¬ 
makers  to  project  the  environment's  status  into  the  future  (Artman,  1998).  They  can  predict 
at  time  t  what  the  situation  will  be  like  at  time  t+1  (Brehmer,  1990).  As  such,  research  on 
the  processes  surrounding  situation  awareness  could  assist  in  a  number  of  areas,  including 
the  design  of  appropriate  decision-making  aids  and  interfaces. 

2.1.5  Previous  Experience  and  NDM 

Related  to  the  development  of  situation  awareness,  is  expertise  (Dreyfus,  1997).  In  a 
summary  of  the  great  military  blunders  throughout  history,  David  (1997)  notes  that  a 
"typical  handicap  seems  to  be  a  lack  of  command  experience.  Naivety  tends  to  promote 
vacillation  and  overcaution,  resulting  in  lost  opportunities  and  ultimately  disaster"  (p.  1). 
NDM  theory  suggests  that  people  draw  on  previous  experience  to  make  sense  of  current 
situations.  Experts  establish  situation  awareness  by  relying  on  familiar  stimulus  cues.  It 
has  been  suggested  that  with  experience,  people  learn  which  cues  to  attend  to,  and  what 
information  to  filter  out  (George,  et  al.,  1996). 

DeGroot  (1965/1978)  and  Chase  and  Simon  (1973)  compared  the  performance  of  chess 
masters  and  novices.  They  found  that  they  did  not  differ  significantly  on  memory  abilities, 
depth  of  planning  or  other  similar  areas.  Instead,  the  difference  seemed  to  be  that  the 
expert  players  could  look  at  the  entire  complex  chess  display  and  condense  it  into 
meaningful  chunks",  where  the  novices  tried  to  understand  the  whole  display.  (Means, 
Salas,  Crandall  &  Jacobs,  1993,  p.  310).  Means  and  Gott  (1988)  suggest  that  expert  and  less 
expert  decision-makers  apply  the  same  rules  but  to  a  different  content.  Less  expert 
decision-makers  may  experience  cognitive  overload  more  rapidly  because  they  attend  to 
all  information,  and  do  not  filter  irrelevant  pieces  (Randel,  Pugh  &  Reed,  1996).  As 
expertise  grows,  individuals  use  tacit  knowledge,  and  see  the  world  through  categories 
(Means,  Salas,  Crandall  &  Jacobs,  1993).  Experts  acquire  the  ability  to  chunk  information. 
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recognise  familiar  patterns,  and  attend  to  critical  indicators.  This  is  referred  to  as  the 
proceduralisation  of  declarative  knowledge.  (Means,  Salas,  Crandall  &  Jacobs,  1993). 
When  a  situation  is  recognised  from  previous  experience,  it  can  directly  stimulate  an 
appropriate  course  of  action  (Rasmussen,  1983). 

A  cognitive  explanation  looks  at  how  previously  solved  problems  are  stored  in  memory 
(reference  problems).  Each  reference  problem  can  be  broken  down  into  three  constitutive 
features: 

•  Objective  features:  This  part  of  the  memory  characterises  the  problem  and  its 
objectives. 

•  Action  features:  This  details  the  steps  that  are  involved  in  solving  the  problem. 

•  Environmental  features:  This  part  compiles  the  cues  that  were  present  in  the  past 
problem  scenario.  These  cues  may  suggest  a  problem  solution  that  is  likely  to  work. 

There  are  also  thought  to  be  levels  of  abstraction  in  memory  involved  in  the  NDM  process. 
The  first  is  concerned  with  concrete  features  (specific  observable  events  or  objects).  The 
second  level  involves  abstract  features  that  help  generalise  the  problem  type  and  solution 
method.  Thirdly  are  environmental  cues  that  may  indicate  a  similar  related  problem. 

This  cognitive  model  allows  recognition-based  solutions  of  new  problems  that  resemble 
but  are  not  necessarily  identical  to  those  previously  experienced  (Cohen,  Freeman  &  Wolf, 
1996).  During  situation  assessment,  the  reference  problems  that  match  the  current 
situation  are  activated.  The  "problem  solution. .  .associated  with  those  in  memory  become 
candidate  actions  "(Noble,  1993,  p.  305). 

Noble  (1993)  states  that  "matching  observed  and  reference  features  may  sometimes  entail 
considerable  sophisticated  information  processing  requiring  general  world  knowledge"(p. 
290).  Frequently,  a  decision-maker  is  forced  to  make  a  decision  with  incomplete 
information.  In  such  circumstances,  the  abstract  level  of  memory  facilitates  NDM.  Lipshitz 
(1989)  found  that  Israeli  army  officers  seemed  to  make  decisions  by  matching  situations  to 
an  associated  action.  The  nature  of  their  experience,  and  consequently  the  patterns 
recognised,  affect  the  decisions  they  make  (Means,  Salas,  Crandall  &  Jacobs,  1993). 

In  summary,  experts  and  novices  differ  in  how  they  use  their  domain  knowledge,  and  not 
in  their  ability  to  use  particular  problem  solving  methods  or  decision  rules  (Means,  Salas, 
Crandall  &  Jacobs,  1993;  Drillings  and  Serfaty,  1997).  Beach  et  al.  (1997)  support  the  idea 
that  decision-making  events  require  decision-makers  to  use  a  "broad  range  of  contextual 
knowledge  in  both  situation  analysis  and  problem  solving  in  order  to  arrive  at  a 
decision" (p.  33).  Therefore,  expertise  is  speculated  to  be  domain  specific,  because  general 
problem  solving  skills  are  not  used.  It  is  also  proceduralised,  meaning  that  decision¬ 
making  is  directly  linked  to  action  and  the  conditions  of  its  applicability.  (Means  et  al., 
1993)  This  may  explain  why  decision-making  can  seem  intuitive  to  the  decision-maker. 


2.2  Distributed  Decision  Making:  NDM  in  a  Team  Environment 

As  well  as  involving  expertise,  many  NDM  tasks  are  too  complex  to  be  handled  by 
individuals.  Instead,  teams  form  to  achieve  goals  beyond  the  reach  of  individual  members 
(Brannick,  Roach  &  Salas,  1993).  Known  as  distributed  decision-making,  the  task  is 
distributed  between  team  members  who  must  have  shared  situation  awareness  and  who 
work  towards  a  common  goal.  Each  person  is  only  required  to  perform  part  of  the  whole 
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task.  This  is  essentially  the  means  by  which  the  Command  and  Control  (C2)  of  a  defence 
force  functions. 

C2  is  the  process  by  which  a  commander  exercises  authority  and  direction  over  the  team 
in  order  to  accomplish  a  specified  mission.  This  is  achieved  by  arranging  personnel, 
equipment,  and  communications  facilities,  as  well  as  using  procedures  that  are  crucial  to 
direct,  control,  coordinate  and  monitor  the  team.  Situation  awareness  in  this  setting  is 
important.  Drillings  and  Serfaty  (1997)  state  that  effective  commanders  are  encouraged  to 
visualise  the  battlefield,  asses  risk,  and  visualise  change  while  forming  plans.  Every  action 
focuses  on  achieving  the  specified  mission  (McCann  &  Pigeau,  1999),  It  follows  that 
achieving  a  good  understanding  of  the  processes  behind  distributed  decision-making  is 
important  in  a  military  setting. 

2.2,1  Communication 

A  fundamental  problem  in  distributed  decision-making  is  knowing  how  best  to  organise 
the  communication  architecture  so  that  team  members  have  shared  situation  awareness 
(Brehmer,  1998).  Stout,  Cannon-Bowers,  Salas  and  Milanovich  (1999)  found  that 
"communication  strategy  alone  affected  the  degree  of  coordinated  performance  attained 
by  teams  during  periods  of  increased  workload"  (p.61).  It  is  particularly  important  in 
hierarchical  organisations  that  the  most  efficient  and  effective  communication  architecture 
is  used,  so  that  orders  can  be  understood  and  thus  successfully  carried  out. 

Artman  (1999)  argues  that  "it  is  important  for  researchers  and  organisations  to  understand 
how  different  information  architectures  affect  the  process  and  outcome  of  controlling 
dynamic  systems"  (p.  1404).  He  examined  parallel  and  serial  communication  architectures 
in  a  hierarchical  organisation.  The  aim  was  to  enhance  teamwork  by  investigating  which 
communication  technique  was  the  most  appropriate  in  a  dynamic  environment.  For 
parallel  communication,  participants  received  the  same  information.  With  serial 
communication,  information  was  filtered  down  a  chain  of  command.  This  contrasts 
information  rich  communication  with  a  limited  window  of  communication.  Artman  also 
examined  fixed  communication  (where  participants  were  told  to  use  a  specific  style),  and 
non-fixed  communication  (where  participants  were  left  to  change  between  styles  as  they 
pleased).  There  was  a  significant  difference  between  fixed  and  non-fixed  communication 
styles,  but  not  between  parallel  and  serial  communication. 

Chapman  (2000)  expanded  on  this  and  examined  open  versus  restricted  communication 
architecture.  In  distributed  decision-making,  co-ordination  of  team  members  is  difficult 
because  everybody  has  a  limited  perspective  of  the  task.  Therefore,  information-rich 
communication,  where  all  information  regarding  the  task  is  revealed  to  every  team 
member,  may  lead  to  a  greater  understanding  of  the  situation.  In  contrast  to  this,  Artman 
(1999)  states  that,  "communication  must  be  relevant  in  order  to  support  team  situation 
awareness"  (p.  1406).  Hearing  all  information  regarding  the  task,  particularly  that  which  is 
not  relevant,  may  lead  to  information  overload  and  cognitive  fatigue.  This  may  decrease 
decision-making  performance. 

2.3  Studying  NDM 

Previous  research  in  NDM  has  focussed  on  several  applied  areas.  For  example,  Klein  et  al. 
(1986)  investigated  decision-making  in  fire  fighters.  Serfaty,  Macmillan,  Entin  and  Entin 
(1997)  carried  out  research  into  the  nature  of  battle  command  decision  expertise.  Fiedler 
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and  Link  (1994)  investigated  leader  intelligence,  interpersonal  stress  and  task  performance 
among  military  officers.  Gaba  (1991)  studied  the  way  anaesthesiologists  make  decisions  in 
teams.  These  types  of  investigation  advance  understanding  of  cognitive  processes,  and  aid 
in  the  design  of  interfaces  and  decision  support  systems.  Other  domains,  such  as  nuclear 
power  plants,  oilrigs  and  aviation  have  also  been  subject  to  NDM  research  (Klein,  1997).  In 
all  of  these  situations,  human  error  can  result  in  death  of  oneself  or  another. 

2.3.1  The  Use  of  Micro  wo  rids  and  Computer  Simulations 

Because  of  its  complexity,  it  is  difficult  to  study  naturalistic  decision-making  in  a  real 
world  environment.  As  a  result,  the  use  of  microworlds  and  other  computer  simulation 
techniques  has  become  prevalent.  A  microworld  is  where  "we  select  the  important 
characteristics  of  the  real  system  and  create  a  small  and  well  controlled  simulation  system 
with  these  characteristics"  (Granlund,  1998,  p.  91).  The  advantage  of  this  type  of  research 
is  that  it  allows  researchers  to  set  a  relevant  complexity  level  in  a  controlled  research 
environment. 

2.3.2  Networked  Fire  Chief 

The  microworld  used  by  Chapman  (2000)  was  a  computer  simulation  called  Networked  Fire 
Chief  (NFC).  It  requires  the  operator  to  make  decisions  under  continually  changing 
conditions.  Participants  are  required  to  fight  fires  that  spontaneously  break  out  on  a  map, 
thus  providing  an  element  of  uncertainty.  Also,  the  tasks  involved  in  successfully  fighting 
a  fire  require  the  co-ordination  and  co-operation  of  a  team  of  people.  An  advantage  of  the 
NFC  program  is  that  it  can  be  networked,  so  that  hierarchical  organisation  structures  can 
be  examined. 

Variations  in  weather  are  represented  by  wind  direction  and  wind  speed.  The  participants 
are  required  to  use  the  appliances  to  extinguish  the  fires.  The  appliances  have  some  of  the 
same  limitations  as  their  real  world  equivalents.  The  decision-maker  is  also  required  to 
prioritise  areas  when  allocating  resources  to  fight  the  fires.  The  order  of  priority 
established  on  NFC  is:  1)  residences,  2)  pastures,  3)  national  park,  and  4)  grassland.  The 
purpose  of  this  variety  in  landscape  is  to  create  a  more  complex  and  realistic  goal. 
Performance  scores  show  the  percentage  of  landscape  left  after  a  designated  period  of  time 
had  elapsed,  taking  land  priority  into  consideration. 

It  should  be  noted  that  NFC  does  not  mirror  everything  that  occurs  in  a  military  context. 
Rather,  it  is  a  tool  used  to  investigate  the  NDM  theory.  It  brings  the  sort  of  variables  into 
play  that  would  be  involved  in  NDM  such  as  uncertainty,  complexity,  and  feedback  loops. 
Currently,  NDM  is  an  umbrella  theory  for  all  situations  displaying  these  characteristics. 
Brehmer  (1994)  noted  that  like  emergency  target  systems,  air  traffic,  and  military  enemy 
force  manoeuvres,  the  forest  fire  is  a  complex,  dynamic,  autonomous  system  (Brehmer, 
1994).  Therefore,  it  should  be  possible  to  refine  the  theory  by  using  any  of  these 
appropriate  contexts.  A  criticism,  however,  is  that  fire  fighting  is  very  different  to  military 
operations.  The  major  way  in  which  it  is  different  is  that  military  personnel  are  working 
with  an  intelligent  enemy,  whereas  fires  do  not  think.  To  that  extent,  fires  could  be  argued 
to  be  slightly  more  predictable.  Therefore,  it  is  important  to  work  towards  finding  whether 
the  results  using  NFC  are  context  specific. 
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2.3.3  The  Use  of  Wargame  Simulations 

The  use  of  wargames  in  military  research  has  started  to  increase.  According  to  Chaloupka, 
Coelho  and  Borges-Dubois  (1991)  "the  importance  of  war  gaming  to  military  planners  lies 
in  its  ability  to  explore  the  dynamic  interactions  of  players".  However,  one  mustbe  critical 
of  their  application  and  be  particular  as  to  what  is  concluded  from  simulation  results. 
Wargaming  is  useful  for  investigating  processes  but  not  calculating  outcomes  (Perla, 
1991). 

Thompson  (1991)  suggests  that  where  wargames  seem  to  mirror  a  real  battle  is  in  the 
decision-making  process.  Players  monitor  the  environment,  make  a  decision,  and  live  with 
it.  Therefore,  similar  to  other  microworlds,  wargame  simulations  should  be  valuable  tools 
for  investigating  NDM.  An  appropriate  wargame  should  create  a  sense  of  danger,  exertion 
uncertainty  and  chance.  The  simulation  used  in  this  study,  Janus,  creates  this,  and  combats 
many  criticisms  that  have  been  made  towards  simulation  wargames. 

2.3.4  Janus:  A  Computer  Wargame  Simulation 

Janus  is  an  interactive  simulation  wargame  that  allows  multi-sided  combat,  under  realistic 
simulation  conditions.  The  view  on  the  screen  is  a  two-dimensional  map  with  grid  lines. 
However,  to  the  weapon  systems  the  terrain  is  three-dimensional  (with  elevation  and 
contour  lines).  This  affects  manoeuvrability  and  line  of  sight.  Unless  the  enemy  is  in  the 
line  of  sight,  each  player  can  only  see  their  own  forces.  When  an  enemy  enters  the  line  of 
sight  of  a  friendly  unit,  it  is  spotted  and  identified.  The  units  then  automatically  engage 
with  the  enemy,  making  decisions  about  what  type  of  ammunition  they  will  use  (this 
decision  is  programmed  into  the  computer's  database). 

Janus  models  real  world  phenomena,  including  natural  and  man  made  obstacles,  the 
requirement  for  route  planning,  direct  and  indirect  fire  engagements,  planned  fire 
missions,  and  tactical  decision-making  as  the  game  progresses.  The  tasks  require  the  co¬ 
ordination  and  co-operation  of  a  team  of  people.  Similar  to  NFC,  Janus  allows  a  team  of 
three  or  more  participants.  One  participant  can  act  as  a  battle  commander,  while  others 
can  act  as  sub-unit  leaders.  This  sets  up  a  requirement  for  distributed  decision-making.  A 
communication  system  can  also  be  used  between  the  team  members. 


2.4  The  Current  Study 

2.4.1  Overview 

As  was  mentioned,  to  achieve  effective  and  efficient  functioning  in  the  Defence  Force,  it  is 
important  for  research  to  be  conducted  that  identifies  the  following  factors:  a)  the 
development  of  expertise;  and  b)  how  communication  facilitates  situation  awareness.  It  is 
also  important  to  identify  an  appropriate  context  for  testing  these  processes,  and  to 
determine  whether  results  obtained  using  a  simulation  war  game,  representing  an  area 
similar  to  the  expert  participants  area  of  experience,  are  consistent  with  those  found  by 
Chapman  (2000)  using  a  context  removed  from  the  expert's  area  of  experience.  In  her 
study.  Chapman  (2000)  investigated  the  effects  of  management  structure,  communication 
architecture  and  previous  experience  (military)  on  NDM  performance  using  NFC.  A 
comparison  was  made  between  information  rich  (open)  and  restricted  communication. 
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•  Open  communication:  All  communications  were  heard  by  all  team  members. 

•  Restricted  communication:  Communication  was  limited  to  the  commander  and  one  of 
his  subordinates  at  any  one  time.  No  one  else  in  the  team  could  hear  the  transmission. 

Chapman  (2000)  also  compared  military  personnel  with  civilian  subjects.  A  limitation  in 
this  study  was  that  the  micro-world  did  not  closely  represent  the  area  in  which  the  expert 
participants  had  gained  their  experience.  Although  past  experience  can  be  used  in  an 
abstract  way,  and  generalised  to  previously  experienced  reference  problems,  it  may  be  the 
case  that  the  difference  in  context  between  a  fire  fighting  scenario,  and  a  military  scenario 
is  too  great.  As  was  mentioned,  a  major  criticism  is  that  military  personnel  are  working 
with  an  intelligent  enemy,  whereas  fires  do  not  think,  and  are  seen  as  less  predictable.  If 
previous  experience  is  context  specific,  testing  should  be  performed  in  an  appropriate 
context. 

As  a  consequence,  the  current  study  will  use  the  computer  war  game  simulation,  Janus.  A 
comparison  will  be  made  between  experienced  participants  (military)  and  inexperienced 
participants  (civilian  DSTO  employees).  Participants  will  be  tested  with  the  two 
communication  architectures,  defined  in  Chapman's  experiment  (2000).  To  investigate  the 
development  of  expertise  in  the  military  and  civilian  participants,  learning  effects  will  be 
monitored.  This  should  also  help  determine  whether  NDM  performance  is  context 
specific. 

2.4.2  Hypotheses 

The  hypotheses  for  the  current  study  are: 

1.  Military  personnel  will  outperform  civilian  subjects. 

2.  Open  communication  architecture  will  facilitate  situation  awareness,  thereby 
producing  higher  performance  outcomes. 

3.  Participants  will  perform  better  in  Janus  trials  as  they  have  more  experience  with  the 
system. 


3.  Method 


3.1  Participants 

Fifteen  participants  took  part  in  the  study.  Their  mean  age  was  40  (sd=8.28).  They 
comprised  two  groups: 

•  Military  personnel  (6  participants  in  2  teams) 

•  Civilian  subjects  (9  participants  in  3  teams) 

The  two  groups  were  divided  into  teams  consisting  of  three  members.  A  description  of 
their  ages  is  shown  in  Table  1.  There  were  fourteen  male  participants,  and  one  female.  The 
civilian  participants  were  recruited  by  advertising  the  experimental  opportunity  on  a 
DSTO  Salisbury  site  mailing  list.  The  Army  personnel  were  recruited  through  Major 
Simon  Harvey.  Interested  respondents  were  scheduled  according  to  availability. 
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Table  1.  Participants'  mean  ages  across  teams  (M=  miLitary  group,  C=  civilian  group) 


Group  1  (M) 

Group  2  (C) 

Group  3  (C) 

Group  4  (C) 

Group  5  (M) 

M 

SD 

M  SD 

M  SD 

M  SD 

M 

SD 

30.67 

6.51 

44.33  2.89 

37.67  11.59 

35.67  8.50 

41.33 

7.57 

3.2  Apparatus 

3.2.1  Information,  Data  Sheets  and  Questionnaires 

•  Information  Sheet:  All  interested  participants  were  sent  an  information  sheet  via  e- 
mail.  This  sheet  provided  a  schedule  for  their  participation  in  the  experiment.  It 
also  outlined  the  study  aims  and  described  the  experimental  procedure  over  the 
three  days  (see  Appendix  A,  A.l). 

•  Demographic  Information  Questionnaire:  This  questionnaire  aimed  to  uncover 
confounding  factors  that  might  influence  results.  It  included  age,  sex  and  job  title, 
previous  experience  with  NDM,  recency  of  this  experience,  use  of  computer 
simulation,  and  amount  of  experience  with  this  simulation  (see  Appendix  A,  A.2). 

•  Notes  to  the  Commander /Sub-unit  Leaders:  These  sheets  outlined  the  roles 
expected  of  the  different  team  members.  It  also  contained  information  specific  to 
roles,  and  rules  of  engagement  (see  Appendix  A,  A.3  &  A.4). 

•  Weapon  Unit  Guide:  This  spreadsheet  gave  information  to  the  participants  on  the 
full  name  of  the  combat  elements  involved  in  the  scenario  (both  blue  force,  and 
suspected  red  force).  It  also  provided  -weapon  range,  sensor  range  and  maximum 
speed  for  each  of  the  units  (see  Appendix  A,  A.5). 

•  "What  weapon  units  look  like  in  Janus"  sheet:  This  illustrated  what  each  of  the 
weapon  units  looks  like  on  the  Janus  screen  (in  a  visual  form). 

•  Casualty  Sheet:  This  sheet  listed  the  weapon  units  controlled  by  each  blue  team 
member,  and  left  a  space  to  mark  down  any  losses  (see  Appendix  A,  A.6). 

•  Radio  Protocol:  The  radio  protocol  outlined  efficient  protocol  for  transmitting 
information  across  a  radio  network.  It  also  listed  team  member  names  and 
experimental  roles  (see  Appendix  A,  A. 7). 

•  Training  Protocol:  The  training  protocol  outlined  the  capabilities  of  Janus,  and 
included  tactical  information  needed  to  make  decisions.  It  described  the 
experimental  architecture,  the  participant's  j  >le  in  the  experiment,  including  the 
functions  they  were  expected  to  carry  out.  It  then  outlined  general  information 
regarding  the  Janus  war  game.  This  included  information  on  game  time,  the  zoom, 
and  the  communication  requirements.  It  then  described  the  terrain  (topographical 
features,  man  made  and  natural  obstacles),  the  movement  capabilities  (route 
planning,  tracked,  wheeled,  foot  and  flyers)  line  of  sight  (obscuration  by  smoke, 
vegetation  and  terrain)  and  direct  and  indirect  fire  engagements  and  planned 
missions  (Appendix  B). 

•  Mission  Planning  Guide:  This  was  a  guide  on  the  basics  of  mission  planning,  based 
on  observa tions  of  a  group  of  three  military  participants  involved  in  planning  for  a 
Janus  scenario.  This  was  written  into  a  suggested  planning  protocol  (see  Appendix 
C). 

•  Intent  Statement:  The  intent  statement  described  the  goals  for  both  of  the  scenarios 
(see  Appendix  D). 

•  Planning  Tools:  A  large  paper  map  of  Kamaria  was  laid  on  a  table  (this 
corresponds  to  the  landscape  represented  on  Janus).  A  plastic  overlay  and  markers 
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were  provided  to  aid  in  route  planning.  A  video  camera  was  used  to  record  all 
planning  sessions. 

•  Communication  Check-sheet:  The  communication  check-sheet  was  used  to  record 
the  frequency  of  desirable  and  non-desirable  events  during  radio  transmissions.  It 
listed  behaviours  that  may  indicate  communication  problems,  and  those  that 
might  assist  in  the  maintenance  and  development  of  situation  awareness  (see 
Appendix  A. 8). 

•  Post  Trial  Questionnaire:  Following  testing,  a  post  trial  questionnaire  was  filled  in 
by  participants.  This  enquired  about  their  changing  perceptions  of  situation 
awareness,  development  of  expertise  and  preference  for  communication 
architecture.  It  used  a  5-point  scale  for  data  encoding.  This  scale  included 
responses  ranging  from  strongly  agree  to  strongly  disagree  (Appendix  E). 

3.2.2  Janus  and  Networked  Computers 

Four  computers  running  Janus  were  networked  in  the  experimental  set  up.  They  were  all 
physically  isolated  from  each  other.  Two  computers  represented  the  units  under  control  of 
the  friendly  team  sub-unit  leaders.  The  enemy  team  used  one  monitor  and  the  remaining 
monitor  was  set  up  as  a  "God  screen".  This  monitor  showed  all  units  from  both  sides, 
providing  an  overall  picture  of  what  was  happening. 

As  well  as  Janus,  the  current  study  used  the  BCSS  and  DICE1.  DICE  acts  to  link  the  action 
happening  on  the  Janus  screen  with  a  BCSS  monitor.  It  allows  the  positioning  of  the 
weapon  units  moving  in  Janus  to  be  represented  on  the  BCSS  monitor.  The  system 
updates  the  positions  of  weapon  units  controlled  by  both  sub-unit  leaders  on  the  BCSS 
map.  Friendly  units  that  were  destroyed  would  cease  to  move  on  the  BCSS  screen. 

Enemy  sightings  did  not  appear  on  the  BCSS  screen.  Rather,  the  commander  needed 
his/her  sub-unit  leaders  to  radio  through  enemy  coordinates.  These  could  then  be 
updated  by  the  commander,  using  markers  on  the  transparency  overlay  covering  the  BCSS 
screen.  There  was  also  a  colour  printout  of  the  Janus  screen  located  below  the  BCSS  screen. 
This  allowed  the  commander  to  be  more  aware  of  what  the  sub-unit  leaders  were  seeing. 

3.2.3  The  Whiteboard /Recording  material 

A  whiteboard  was  provided  for  the  commander  to  use  if  desired.  In  addition,  all 
participants  were  supplied  with  paper  and  pens  to  copy  down  any  transmissions  they  felt 
necessary. 

3.2.4  Communication  System 

Participants  communicated  under  both  communication  architectures  using  a  set  of  radios. 
These  could  be  used  on  the  same  frequency  to  simulate  an  open  communication,  or  two 
different  frequencies  to  simulate  restricted  communication.  During  restricted 
communication,  the  commander  would  have  two  radios  on  different  frequencies  to  pick 
up  the  two  sub-unit  leaders.  High  level  commands  were  issued  by  the  commander  who 
had  a  picture  on  BCSS  of  where  all  of  the  blue  forces  were.  Feedback  was  required  from 
the  sub-unit  leaders. 


1  Distributed  Interactive  C3I  Effectiveness  simulation  capability. 
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The  experimenters  heard  all  transmissions  by  team  members  through  the  master  radios. 

3.2.5  Scenarios 

Two  scenarios  of  equivalent  complexity  were  used.  Both  scenarios  had  the  same  weapon 
units,  but  were  set  in  different  areas.  One  was  used  for  open  communication,  while  the 
second  was  used  for  restricted  communication.  This  was  alternated  for  teams. 

The  scenarios  ran  for  1%  hours,  and  were  set  in  the  fictional  land  of  Kamaria.  Participants 
were  instructed  to  attempt  to  accomplish  the  goal  specified  in  the  intent  statement. 

3.2.6  The  Interactors 

Because  of  time  constraints,  training  each  participant  to  use  Janus  was  not  possible. 
Therefore,  Janus  interactors  were  trained,  and  used  to  implement  commands  given  by  the 
sub-unit  leaders.  The  interactor  linked  the  participant  to  the  action  happening  on  the 
computer  screen.  This  interactor  was  responsible  for  translating  any  of  their  participant's 
commands  to  the  weapons  units  on  the  Janus  screen.  The  interactors  were  instructed  to  act 
efficiently  on  commands  given  to  them  by  the  participants,  and  to  explain  any  information 
regarding  the  useability  of  Janus.  They  were  instructed  not  to  act  on  their  own  initiative, 
or  make  any  tactical  suggestions  that  may  influence  the  overall  performance  scores. 

To  assess  whether  this  was  adhered  to,  the  experimenters  used  a  check-sheet  to  record 
desirable  and  non-desirable  interactor  behaviour.  This  data  was  used  to  determine 
whether  interactors  were  non-discriminatory,  or  whether  they  actively  participated  in  the 
experiment  and  possibly  influenced  results  (Appendix  A.9). 

3.3  Procedure 

3.3.1  Participant  Groups 

Participants  were  randomly  allocated  into  groups  of  three  (within  their  level  of  expertise 
groups).  One  team  member  acted  as  the  commander,  while  the  other  two  were  his/her 
subordinate  sub-unit  leaders  (one  was  responsible  for  the  ground  force  and  the  other  for 
the  support  force).  The  groups  of  three  people  in  this  study  represent  a  simplistic 
hierarchy,  similar  to  that  used  in  NFC.  Roles  within  each  team  were  randomly  allocated. 

3.3.2  Pre-Trial  Information 

At  the  start  of  the  experimental  session,  participants  were  given  a  folder  containing  an 
information  sheet,  a  demographic  information  questionnaire,  a  Janus  training  protocol,  an 
information  sheet  for  their  designated  role  in  the  experiment,  a  casualty  sheet,  a  weapon 
unit  guide,  a  radio  protocol,  and  a  "what  weapons  look  like"  diagram. 

3.3.3  Training 

After  reading  the  required  information,  each  team  took  part  in  a  training  session  to 
familiarise  them  with  the  specialised  set  up  of  Janus.  Firstly,  participants  were  asked  to 
read  through  the  training  protocol.  They  were  then  taken  through  a  practical 
demonstration  of  the  program  that  closely  followed  the  training  protocol.  During  this 
time,  any  questions  were  answered  to  make  the  participants  confident  that  they  had 
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enough  knowledge  to  effectively  command  an  interactor  to  perform  actions  using  the 
Janus  war  game.  This  practical  demonstration  lasted  approximately  45  minutes. 
Civilian  participants  were  also  given  a  lesson  in  the  basics  of  mission  planning.  This  was 
done  by  means  of  a  pamphlet  explaining  the  phases  used  by  the  military  in  planning 
exercises. 

3.3.4  The  Trials 


The  schedule  for  each  of  the  8  trials  is  illustrated  in  Figure  3. 


Figure  3.  The  schedule  for  the  wargaming  trials 


Once  training  was  complete,  participants  went  through  three  days  of  Janus  trials.  This 
equated  to  8  trials,  each  lasting  up  to  VA  hours.  The  criteria  for  ceasing  a  trial  was  one  of: 

1.  The  mission  was  accomplished  (i.e.  all  enemies  were  defeated,  and  the  goal  was 
reached); 

2.  One  of  the  teams  surrendered;  or 

3.  Game  play  had  lasted  VA  hours  (this  length  of  time  was  needed  for  Janus  to  generate 
meaningful  outcome  data). 

To  control  for  learning  effects,  the  two  scenarios  were  alternated  for  each  trial.  Out  of  the 
eight  trials,  four  were  performed  with  each  communication  architecture.  To  control  for 
order  effects,  each  day  and  team  started  with  different  a  communication  architecture  than 
the  previous  day  or  team. 

3.3.5  The  Planning  Stage 

In  this  phase,  participants  were  read  the  intent  statement.  They  were  then  given  20 
minutes  to  work  together  to  plan  their  mission.  To  assist  with  planning  routes  and  tactics, 
they  were  provided  with  a  map  of  Kamaria,  a  plastic  overlay,  and  some  transparency 
markers. 

3.3.6  The  Simulation  Stage 

Following  planning,  participants  performed  the  wargame.  Interactors  were  introduced  to 
participants  and  team  members  were  physically  isolated  from  each  other.  A  radio  check 
was  then  performed  to  ensure  that  all  radios  were  working  correctly. 

Each  participant  was  positioned  in  front  of  a  computer  screen  showing  a  map  of  Kamaria, 
with  weapon  units  ready  to  fight.  Participants  were  given  5-10  minutes  before  beginning 
game  play  for  instructing  their  interactor  on  how  to  organise  the  movement  routes  of  these 
weapon  units. 

During  game  play,  the  experimenters  did  three  things: 

1.  Observed  interactor /participant  interactions  (filling  in  check-sheets); 
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2.  Watched  the  overall  battle  picture  on  the  Janus  screen  and  made  observational  notes; 
and 

3.  Listened  to  the  radio  transmissions  and  filled  in  communication  check-sheets, 

3.3.7  Post-Trial  Procedure 


At  the  end  of  the  trial,  participants  were  invited  to  the  planning  room  to  discuss  their 
battle.  No  limits  were  put  on  this  discussion.  After  the  final  trial,  they  were  given  the 
post-trial  questionnaire. 


4.  Results 

It  should  be  noted  that  because  this  is  such  a  new  area  of  research,  the  current  study  is 
largely  exploratory.  Rather  than  supplying  definitive  conclusions,  it  should  be  valuable  in 
providing  insights  into  future  research  areas  and  methods. 

As  was  mentioned,  the  design  of  the  experiment  was  based  on  eight  trials  (four  trials 
under  each  communication  condition)  (see  Table  2).  However,  due  to  time  and  resource 
constraints,  an  equal  number  of  trials  for  each  group  was  not  possible.  All  three  of  the 
civilian  groups  participated  in  eight  trials.  One  military  group  was  involved  in 
experimental  refinement,  so  their  results  have  been  excluded  from  final  analysis.  Due  to 
time  constraints,  the  second  military  group  participated  in  only  four  trials. 

Due  to  small  sample  size  and  unequal  groups,  the  following  is  a  descriptive  analysis  of 
results.  It  should  reveal  trends  that  could  be  pursued  in  further  research,  rather  than 
finding  significant  effects. 

Table  2.  The  experimental  design 


Military 

Civilian 

Open  communication 

2  trials 

4  trials 

Restricted  communication 

2  trials 

4  trials 

4.1  Description  of  Participants 

The  demographic  questionnaire  revealed  individual  differences  as  shown  in  Table  3. 


Table  3.  Individual  differences 


Frequency  of  Positive 
Response 

Percent  of  Total 

Have  you  previously  used  Janus? 

4 

23.5% 

Have  you  worked  with  your  team  mates  before? 

6 

35.3% 

Have  you  ever  played  a  computer  game  or 
simulation? 

14 

82.4% 
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Among  those  who  had  previously  used  a  computer  game  or  simulation,  the  frequency  of 
usage  was  generally  not  high.  Table  4  indicates  the  amount  of  simulation  exposure 
participants  have  had  before  commencing  the  trials.  This  table  shows  that  out  of  the 
participants  who  answered  affirmatively  to  having  previously  played  a  computer  game, 
the  mean  amount  of  time  since  they  had  played  was  approximately  2  years.  The  majority 
(35.3%)  of  respondents  indicated  that  they  had  played  for  more  than  three  hours. 
However,  the  infrequency  of  their  game  or  simulation  usage  indicates  that  their  expertise 
in  game  play  would  be  minimal. 


Table  4.  Exposure  to  computer  simulations  or  games 


Minimum 

Maximum 

Mean 

SD 

(days) 

(days) 

(days) 

(days) 

When  was  the  last  time  you  played  a  computer 

game  and/or  simulation? 

0 

4745 

741 

1344.31 

When  was  the  time  before  that  you  played  a 

computer  game? 

1 

1095 

200.54 

339.23 

Frequency  of  Response  (%) 

Less  than  30 

30  mins  to  1 

1  to  3  hours 

More  than 

mins 

hour 

three  hours 

How  long  did  you  spend  on  your  last  game  or 

simulation? 

0 

23.5 

23.5 

35.3 

How  long  did  you  spend  on  the  game  or 

simulation?  (the  time  before) 

17.6 

11.8 

29.47 

17.6 

4.2  Janus  Performance  Scores 

To  assess  participant  performance,  loss  exchange  ratio  (enemy  kills  were  divided  by 
friendly  losses)  was  used.  This  generated  a  performance  score  relative  to  that  of  the 
enemy.  This  also  allowed  enemy  performance  over  time  to  be  monitored. 

4.2.1  Communication  architecture 

Table  5  shows  the  effect  of  communication  architecture  on  loss  exchange  ratio.  In  line  with 
the  hypothesis  that  open  communication  architecture  would  facilitate  situation  awareness, 
open  communication  has  produced  higher  performance  scores. 

Table  5.  Communication  architecture  and  the  influence  on  Janus  Kill/Loss  ratios 


Mean 

Standard 

Deviation 

Max 

Open 

2.04 

1.26 

5.60 

Restricted 

1.44 

0.79 

2.67 

4.2.2  Previous  Experience 

Table  6  shows  a  substantial  difference  between  the  military  personnel  and  the  civilian 
participants.  This  supports  the  hypothesis  that  military  personnel  would  outperform 
civilian  subjects. 
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Table  6.  The  effect  of  previous  experience  on  Janus  scores 


Mean 

Standard 

Deviation 

Max 

Military 

2.82 

1.93 

5.60 

Civilian 

1.56 

0.80 

3.19 

Overall 

1.72 

1 

5.60 

4.2.3  Performance  Measures  Across  Groups 

Table  7  shows  that  although  military  personnel  achieved  higher  performance  scores  on  the 
current  Janus  simulations,  there  was  variation  between  the  civilian  groups'  performance 
scores. 

Table  7,  Janus  performance  across  groups 


Group  2 

Group  3 

Group  4 

Group  5 

Mean 

1.87 

1.4 

1.42 

2.82 

Standard  Deviation 

0.53 

1.13 

0.66 

1.93 

Maximum  Score 

2.67 

3.19 

2 

5.60 

4.2.4  Development  of  Expertise  Across  Trials 

Figures  4  and  5  show  mean  performance  scores  across  trials.  Neither  graph  seems  to 
indicate  a  straightforward  development  of  expertise.  Performance  seems  to  increase  over 
time.  However,  there  is  no  meaningful  linear  or  quadratic  function  to  illustrate  this.  Dips 
in  performance  are  found  at  trials  2, 4,  and  7.  During  Days  1  and  3,  participants  performed 
better  in  their  initial  trials  each  day.  In  addition,  the  second  day  of  participation  produced 
higher  performance  scores  than  the  first. 


Run  Order 


Figure  4.  The  average  performance  of  civilians  across  trials 


Mean  Performance  Mean  Performance 
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Figure  6. 


The  development  of  expertise  over  trials  for  military  participants 


The  performance  for  each  group  across  the  eight  trials 
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4.2.5  Enemy  Performance  Over  Time 

Enemy  performance  seemed  to  stay  relatively  constant  across  trials.  Three  peaks  on  Figure 
7  indicate  scenarios  where  the  enemy  had  few  losses. 
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Figure  7.  The  enemy's  performance  over  time 


4.3  Communication 

There  was  only  minor  variation  in  communication  across  groups.  Table  8  shows  that  the 
most  noticeable  differences  were  a  larger  number  of  acknowledgments  and  updates  made 
by  military  personnel,  while  they  made  less  requests  for  information.  Another  noteworthy 
result  was  that  the  provision  of  information  was  less  than  the  requests  for  each  group.  The 
military  group  seems  to  have  the  closest  relationship  between  information  request  and 
information  feedback. 


Table  8.  Communication 


Communication  Action 

Mean  Number 
Group  2(C) 

Group  3(C) 

Group  4  (C) 

Group  5(M) 

Acknowledge 

N/A 

25 

22 

40.75 

Provide  updates 

30.25 

31 

19.2 

49 

Identify  Actions  that  Need  Correcting 

7.75 

7 

9.2 

7 

Request  information 

37.5 

38.4 

16 

19 

Provide  Requested  Information 

31.7 

32.6 

11.2 

18.5 

Identify  actions  that  need  correcting 

7.75 

7 

9.2 

7 

Identify  problems  to  task  completion 

7 

11.4 

6 

5.25 

4.4  Self-Perception 

4.4.1  Situation  Awareness 

Figures  8  and  9  show  participants'  perception  of  their  changing  situation  awareness.  All 
participants  except  one  felt  that  their  situation  awareness  increased  after  the  first  day.  This 
was  true  for  both  military  and  civilian  participants.  However,  no  significant  correlations 
were  found  between  participants'  performance  score  and  beliefs  about  their  development 
of  situation  awareness. 
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Figure  8.  Perception  of  situation  awareness  for  military  personnel 
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Figure  9.  Perception  of  situation  awareness  for  civilians 

4.4.2  The  Development  of  Expertise 

Four  questions  m  the  post-trial  questionnaire  assessed  whether  participants  thought  they 
were  developing  expertise  in  the  task.  The  responses  to  these  questions  ranged  from 
isagree  to  strongly  agree  on  the  five-point  scale.  The  majority  of  participants  tended  to 
agree  or  strongly  agree  with  these  statements.  For  example,  at  the  end  of  the  trials,  100%  of 
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military  participants  strongly  agreed  that  they  had  become  more  skilled  commander /sub¬ 
unit  leaders  under  these  wargame  conditions.  Of  the  civilian  participants,  66.6%  strongly 
believed  this.  There  also  seemed  to  be  a  general  trend  for  participants  to  agree  to  a  higher 
extent  with  these  statements  across  trials.  However,  no  significant  correlations  were  found 
between  participants'  performance  score  and  beliefs  about  their  development  of  expertise. 

4.4.3  Communication  Architecture  and  Situation  Awareness 

The  majority  of  participants  believed  that  open  communication  improved  their  situation 
awareness  and  performance.  After  the  first  two  trials,  70.6%  of  participants  preferred  open 
communication,  5.9%  preferred  restricted,  and  5.9%  could  not  decide.  At  the  end  of  the 
three  days  (8  trials),  77%  preferred  open  communication  and  22%  of  people  could  not 
decide.  No  participants  preferred  restricted  communication.  This  preference  for  open 
communication  is  consistent  with  the  finding  that  open  communication  produces 
significantly  higher  performance  scores. 
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Communication  Architecture  Enabling  Better  Performance  1 
Figure  10.  Preferred  communication  architecture  after  Day  One 
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Communication  Architecture  Enabling  Better  Performance  3 
Figure  11.  Preferred  communication  architecture  after  Day  Three 

4.4.4  Effectiveness  of  Training  Regime 

The  training  regime  seems  to  have  been  relatively  effective.  From  the  post-trial 
questionnaire,  53.4%  of  participants  agreed  or  strongly  agreed  that  they  understood  Janus 
capabilities  from  the  training.  86.7%  of  participants  responded  that  they  understood  the 
experimental  expectations  from  the  supplied  information  (See  Appendix  B). 

4.4.5  The  Interactor/Participant  Interaction 

Participants  indicated  that  interactors  played  an  important  part  in  the  outcome  of  the 
trials.  According  to  check-sheets,  85%  of  what  interactors  did  was  in  compliance  with  the 
participant's  command.  This  included  explaining  capabilities,  acting  on  command,  and 
clarifying  commands.  This  would  suggest  that  interactors  would  not  have  negatively 
influenced  the  scores.  In  addition,  out  of  a  mean  number  of  moves  of  382.5,  the  level  made 
by  each  interactor  was  relatively  similar,  ranging  from  10.45%-  9.21%.  This  represents  an 
even  number  of  commands  issued  across  trials. 

3.22%  of  good  relationship  indicators  were  witnessed  (eg.  encouragement  and  friendly 
comments).  The  remaining  11.88%  of  actions  that  may  have  influenced  results  include 
such  things  as  delayed  response  (1.2%),  incorrect  response  (1.9%),  tactical  suggestions 
(1.89%),  ignoring  commands  (0.2%),  and  acting  on  their  own  initiative  (6.5%).  One 
interactor  was  responsible  for  37.11%  of  the  incidences  of  using  one's  own  initiative  to 
play  the  wargame.  The  same  interactor  was  also  responsible  for  71.42%  of  the  tactical 
suggestions  made.  A  comment  made  by  the  participant  regarding  the  interactor  was: 

My  interactor  took  over  when  he  got  bored.  I  found  this  very  frustrating  and  felt  that  I 
could  have  gotten  more  out  of  the  exercise  if  he  let  me  make  my  own  mistakes  at  the  start. 

. .  .It  took  a  while  to  learn  what  was  what.  I  still  don’t  understand  it  all.  I  need  to  be  able 
to  make  more  mistakes. 
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Two  interactors  stood  out  with  the  amount  of  explanations  offered.  These  percentages 
were  37.67%  and  24.43%  compared  to  the  other  interactors  who  ranged  from  3.3%  to 
16.17%.  Only  5  interactors  responded  incorrectly  to  commands.  Out  of  this  group,  one  was 
responsible  for  69.62%  of  the  incorrect  responses  made. 


5.  Discussion 


This  study  used  the  computer  simulation  wargame  Janus  to  investigate  the  effect  of 
communication  architecture,  and  previous  experience  on  NDM.  The  development  of 
expertise  was  also  monitored  across  trials.  The  current  study  has  proven  successful  in 
identifying  issues  that  need  consideration  when  designing  experiments  on  naturalistic 
decision-making.  It  also  revealed  trends  that  can  be  pursued  in  future  research. 

5.1  The  Effect  of  Communication  Architecture  on  Janus  Performance 

The  data  revealed  a  difference  between  the  two  communication  architectures,  with  open 
communication  producing  higher  performance  scores.  This  supports  the  hypothesis  that 
open  communication  facilitates  situation  awareness.  This  result  is  also  consistent  with 
Chapman  (2000),  where  Networked  Fire  Chief  was  the  experimental  tool.  This  provides 
some  validation  of  the  use  of  Networked  Fire  Chief  experiments  to  develop  the  NDM  theory 
in  relation  to  the  military.  For  future  research,  it  is  recommended  that  this  comparison  be 
investigated  with  larger  participant  numbers.  Forming  larger  teams  could  also  be 
beneficial,  as  it  may  be  the  case  that  with  larger  hierarchies,  open  communication  may  not 
be  as  effective,  as  members  would  have  to  filter  more  information. 

5.2  The  Effect  of  Previous  Military  Experience  on  Janus  Performance 

A  difference  was  found  between  subject  groups,  with  military  participants  outperforming 
civilians.  Because  the  difference  was  substantial,  this  would  be  an  important  comparison 
for  future  research.  Flowever,  it  is  important  to  treat  this  data  with  some  caution.  The 
unbalanced  design,  with  one  military  group  compared  to  three  civilian  groups,  as  well  as 
the  unbalanced  number  of  trials  between  civilian  and  military  groups,  could  have 
produced  this  result.  These  issues  need  to  be  addre-  <ed  in  future  studies. 

It  is  also  necessary  for  future  research  to  more  clearly  define  what  constitutes  expertise. 
According  toRandel,  Pugh,  and  Reed  (1996)  an  expert  is  defined  in  terms  of  the  following 
qualities: 

•  Factual  knowledge; 

•  Efficient  retrieval  of  stored  knowledge; 

•  Ability  to  monitor  motor  and  perceptual  processes;  and 

•  Ability  to  efficiently  allocate  cognitive  resources. 

The  experts  used  in  this  study  were  all  military  officers.  However,  they  indicated  that  they 
were  only  involved  in  combat  training  once  per  year.  In  addition,  their  roles  in  the  military 
were  not  directly  related  to  weapon  unit  control.  Taking  the  above  definition  of  expert 
into  account,  it  should  prove  valuable  to  use  military  participants  who  are  actively 
involved  in  the  areas  in  the  wargame  scenario.  It  would  be  expected  that  because  of  the 
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clear  difference  found  between  groups  in  this  study,  recruiting  participants  who  fit  the 
definition  of  expert  more  closely  should  increase  the  difference  in  performance  between 
groups. 

5.3  The  Development  of  Expertise  Across  Trials 

5.3.1  Experimental  Participants 

No  simple  trend  was  revealed  in  terms  of  the  development  of  expertise  across  trials.  The 
civilian  sample  showed  a  broad  range  of  variability  in  performance  across  their  trials. 
However,  isolating  the  first  and  last  trials  reveals  an  increase  in  performance.  The 
variability  of  performance  in  the  other  trials  results  could  be  partially  accounted  for  by 
fatigue.  The  participants  commented  that  three  trials  in  one  day  left  them  exhausted.  One 
commented  that  "by  the  third  scenario  on  each  day  I  think  there  tended  to  be  a  big  loss  of 
interest  and  it  was  much  more  difficult  to  keep  focused." 

Observations  during  trials  also  showed  that  participants'  enthusiasm  for  the  task  was 
reduced  after  a  number  of  trials.  While  this  varied  from  group  to  group,  the  second  or 
third  scenarios  being  run  each  day  was  generally  affected. 

For  future  research,  it  would  be  beneficial  to  try  to  reduce  participant  fatigue  and  maintain 
their  interest  in  the  task.  This  proves  difficult  when  running  experiments  where  large 
numbers  of  trials  are  needed  to  generate  meaningful  outcomes.  The  balance  between  ideal 
experimental  design  and  participant  willingness  and  availability  has  to  be  taken  into 
consideration. 

5.3.2  Enemy  Performance 

Enemy  performance  over  time  did  not  seem  to  improve,  with  performance  remaining 
relatively  constant.  Three  peaks  of  performance  stand  out  that  could  be  accounted  for  by 
anomalies  in  the  Janus  recording  log  and/or  chance  occurrences. 

It  is  important  for  an  experimental  design  such  as  this  to  have  an  intelligent  enemy,  who 
can  react  to  moves  made  by  the  participants.  At  the  same  time  enemy  performance  must 
be  held  as  constant  as  possible,  so  as  not  to  confound  participants'  results  across  time. 
Overall,  the  current  study  achieved  this  result. 

5.4  Communication 

Military  participants  used  substantially  more  acknowledgments  and  update  reports  than 
civilians.  This  type  of  communication  helps  to  facilitate  teamwork  and  situation 
awareness.  This  could  partially  explain  the  overall  difference  in  performance  scores 
between  military  and  civilian  participants.  Unlike  the  civilians,  military  personnel  also 
requested  information  and  provided  requested  information  at  close  to  the  same  frequency. 
This  is  what  you  would  expect  if  each  request  were  given  a  response.  A  related  issue  was 
the  high  level  of  information  requests  from  some  of  the  civilian  participants.  This  further 
demonstrates  their  lack  of  expertise  in  the  area.  In  particular,  it  is  likely  that  they  did  not 
know  which  information  to  filter  out,  so  were  requesting  more  than  was  necessary  for 
optimal  performance. 
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5.5  Self-Perception 

The  validity  of  the  self-reports  in  the  current  study  can  be  questioned  due  to  the  fact  that 
the  majority  (excluding  the  participants'  perceptions  about  communication  architecture 
and  its  effectiveness)  did  not  significantly  correlate  with  performance  scores.  However,  it 
is  anticipated  that  by  refining  techniques  of  extraction,  more  reliable  self-perception  data 
can  be  obtained.  In  addition,  larger  participant  numbers  would  help  to  consolidate  this 
issue. 

5.5.1  Self-Perception  of  Situation  Awareness 

In  the  civilian  sample,  there  was  a  perception  of  increasing  situation  awareness  from  day  1 
to  day  3.  However,  Group  2  reported  higher  situation  awareness  on  day  2,  compared  with 
day  3.  This  may  be  due  to  different  interactors  giving  varying  quality  in  support.  The 
remaining  two  groups  reported  approximately  the  same  levels  of  situation  awareness 
across  day  two  and  three. 

Two  members  of  the  military  sample  agreed  that  their  situation  awareness  increased  over 
trials,  while  the  third  military  participant  believed  that  his  situation  awareness  decreased 
across  days.  To  clarify  this  issue,  it  needs  to  be  tested  with  larger  participant  numbers. 
Also,  to  obtain  a  more  accurate  measure,  a  different  method  could  be  used  to  measure 
participants'  self-perception  of  situation  awareness.  The  current  study  used  a  5-point  scale, 
ranging  from  strongly  disagree  to  strongly  agree.  Most  participants  tended  to  agree  or 
strongly  agree.  It  may  be  more  useful  to  measure  the  degree  to  which  they  felt  their 
situation  awareness  had  improved  via  a  short  interview  after  each  trial. 

5.5.2  Self-Perception  of  Expertise  Development 

100%  of  military  personal  and  66.6%  of  civilians  strongly  agreed  they  were  becoming  more 
skilled  commander/ sub-unit  leaders  under  these  wargame  conditions.  However,  the  data 
on  expertise  development  showed  both  increases  and  decreases  in  performance.  It  is 
possible  that  participants  were  trying  to  please  the  experimenter  by  answering  in  a 
manner  expected.  Future  studies  should  refine  statistical  performance  measures  of 
expertise  development  to  consolidate  conclusions. 

5.5.3  Self-Perception  of  Communication  Architecture 

The  majority  of  participants  preferred  open  communication  to  restricted  communication. 
One  of  the  civilian  participants  commented;  "being  aware  of  what  the  commander  and 
other  commander  (sub-unit  leader)  were  communicating  about  helped  my  situational 
awareness".  Another  participant  stated  that,  "restricted  communication  introduced  a 
significant  delay  in  requesting  artillery  support.  This  made  estimating  where  to  order 
strikes  on  moving  targets  much  more  difficult."  Thus,  participants  recognised  that  their 
performance  was  improving  with  open  communication  architecture. 

5.6  Training 

The  training  protocol  used  in  the  current  study  seems  to  have  been  effective  in  giving 
participants  a  suitable  level  of  information  about  the  Janus  system.  The  majority  of 
participants  agreed  or  strongly  agreed  that  they  understood  Janus  capabilities.  However, 
during  the  trials,  interactors  provided  a  large  amount  of  capability  explanations.  This 
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could  have  slowed  down  the  decision-making  process  and  resulted  in  possible  loss  of 
weapon  units  in  the  Janus  simulation. 

For  future  investigations  it  is  recommended  that  the  training  regime  be  reviewed,  and 
tested  on  mock  subjects.  Clarity  of  expression,  amount  of  information,  necessity  of 
information,  and  time  allocated  for  training  should  be  considered.  The  method  of 
presentation  should  also  be  reviewed.  In  the  current  study,  participants  were  asked  to 
read  through  the  training  protocol.  Following  this,  a  Janus  expert  took  them  through  a 
practical  demonstration.  During  this  time,  questions  were  answered.  This  occurred 
directly  before  the  first  trial.  If  the  training  occurred  on  a  separate  day,  and  a  short  practice 
session  was  included,  participants  may  feel  more  comfortable  with  the  system  capabilities. 
This  may  improve  their  performance  in  initial  trials,  and  remove  Janus  system  learning 
effects.  A  consideration,  however,  is  the  required  time  commitment  for  participants.  This 
would  effectively  add  another  morning  or  afternoon  to  the  testing  period. 

5.7  Interactors 

In  this  kind  of  investigation  where  humans  are  used  as  experimental  aids,  a  certain  level  of 
control  must  be  kept.  The  results  indicate  that  the  majority  of  interactor  time  was  spent 
performing  actions  to  facilitate  experimental  outcomes.  However,  a  number  of  issues  arose 
that  need  to  be  addressed.  Firstly,  the  relationship  between  the  interactor  and  the 
participant  clearly  varied  from  day  to  day  as  different  interactors  were  scheduled  on. 
Some  interactors  merely  acted  on  commands.  Others  would  laugh  at  participant's  jokes, 
share  smiles,  gossip  between  game  play  or  make  suggestions.  This  may  have  influenced 
performance  across  trials. 

In  addition,  some  interactors  would  indirectly  make  suggestions.  For  example,  they  would 
ask  "Would  you  like  me  to...?"  or  "What  do  you  want  to  do  with  X  ?".  This  implies  that  an 
action  should  be  taken,  but  not  which  action.  This  may  have  prompted  participants  and 
given  them  an  advantage  compared  with  those  where  the  interactor  did  not  indirectly 
suggest  actions. 

A  more  in-depth  training  scheme  should  also  be  established  to  familiarise  interactors  with 
Janus.  Several  interactors  stated  that  they  did  not  feel  confident  with  the  system  at  the 
beginning  of  trials.  This  could  be  combated  with  more  intense  training.  Also,  a 
requirement  for  interactor  screening  should  be  put  in  place  before  the  trials  begin.  As 
mentioned  in  the  results,  there  were  interactor  differences  in  terms  of  using  their  own 
initiative  instead  of  following  orders,  and  making  tactical  suggestions.  A  participant 
affected  by  this  became  frustrated  by  the  end  of  the  day  and  commented  that  "it  took  a 
while  to  leam  what  was  what.  I  still  don’t  understand  it  all.  I  need  to  be  able  to  make  more 
mistakes".  This  might  have  influenced  learning  and  the  development  of  expertise  over 
time  as  this  interactor  was  scheduled  on  the  first  day  of  the  team's  trials,  where  it  would 
be  most  important  for  the  participants  to  try  the  program  and  make  a  few  mistakes  in 
order  to  leam. 

5.8  Limitations 

5.8.1  BCSS  Usage 

The  current  study  used  BCSS  as  a  situation  awareness  tool  for  the  commander.  It  provided 
updates  of  the  friendly  force  positions  at  regular  intervals.  A  BCSS  interactor  was  not  used 
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in  the  current  study  because  of  time  and  resource  constraints.  Commanders  were 
provided  with  transparencies  over  the  BCSS  screen  on  which  to  mark  anything  they 
required  (enemy  position,  artillery  zones).  There  were  several  problems  with  this  setup. 
For  example,  when  friendly  units  were  destroyed  on  the  Janus  screen,  they  stopped 
moving  on  the  BCSS  screen  but  were  still  present.  Commanders  reported  that  they  found 
this  confusing.  This  problem  could  be  combated  in  future  research  by  having  a  trained 
BCSS  interactor  who  could  remove  weapon  units  reported  as  lost. 

A  second  problem  was  that  the  screen  could  not  show  a  large  zoom  and  still  show  all  the 
weapons  units  at  the  same  time.  There  was  also  a  lack  of  correlation  with  the  map  found 
on  the  BCSS  monitor  and  that  in  Janus.  Land  types  seemed  not  to  match  between  maps.  It 
is  important  that  a  high  correlation  between  maps  is  present;  otherwise  effective 
communication  is  more  difficult, 

5.8.2  Scenario  Development 

During  scenario  development,  the  friendly  force  was  divided  into  a  ground  force  and  a 
support  force.  This  created  a  cooperative  situation  that  could  have  reproduced  the  same 
kinds  of  communication  requirements  experienced  in  Networked  Fire  Chief.  To  a  certain 
extent  this  seemed  successful.  However,  the  simplicity  of  the  scenario  was  of  concern,  as 
was  the  lack  of  wide  variation  in  the  performance  scores. 

The  clear  difference  between  the  power  of  the  red  force  compared  with  the  blue  force 
meant  that  the  enemy  had  many  problems  posing  an  effective  attack.  This  is  an  internal 
programming  aspect  of  Janus.  It  may  be  that  in  order  to  create  a  fairer  battle,  the  enemy 
needs  a  greater  number  of  units.  This  would  possibly  create  a  greater  distribution  in 
performance  scores. 

The  scenario  could  also  be  made  more  complex  by  having  multiple  sub-unit  leaders,  each 
of  whom  would  be  in  charge  of  certain  weapon  units.  This  would  allow  use  of  larger 
teams. 

Another  aspect  of  the  scenario  relates  to  the  duration  of  the  trial.  VA  hours  was 
recommended  by  a  Janus  expert  as  an  appropriate  length  of  time  to  draw  meaningful 
results.  During  the  trials,  one  of  three  things  tended  to  happen: 

1.  The  team  defeated  all  of  the  enemy's  ground  units,  and  spent  the  remaining  time 
searching  for  artillery  (usually  fruitlessly); 

2.  The  team  surrendered  because  they  felt  they  had  been  defeated;  or 

3.  Both  teams  had  reached  a  town  and  they  spent  the  remaining  time  scanning  through 
this  area  of  extremely  low  visibility. 

The  third  response  is  possibly  an  artefact  from  the  extremely  low  visibility  in  the  urban 
zones  of  Janus.  This  caused  difficulty  locating  enemy  units  within  the  urban  zone. 
Therefore  time  spent  searching  was  lengthened,  whereas  actual  engagement  with  enemy 
units  was  almost  impossible. 

With  a  more  complex  scenario  and  a  larger  red  force,  the  battle  tactics  and  time-pressured 
decision-making  may  continue  until  the  end  of  the  trial. 
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It  should  also  be  noted  that  in  this  type  of  exercise,  it  is  more  important  to  generate 
accurate  performance  measures  than  for  the  participants  to  achieve  their  goal.  Therefore, 
a  scenario  with  a  higher  level  of  complexity  amongst  the  blue  force,  as  well  as  equal 
distribution  of  power  across  the  blue  and  red  forces,  is  recommended.  It  should  also  be 
pointed  out  to  participants  that  the  intent  statement  is  only  a  guide,  and  that  it  is 
acceptable  if  they  do  not  achieve  the  goal  within  the  allocated  time. 

5.8.3  Respect  for  the  commander 

A  problem  in  the  current  study  was  the  limited  use  of  the  commander  across  groups.  The 
scenario  would  often  be  played  out  without  many  interactions  with  the  commander. 
Because  of  the  random  allocation  of  roles,  the  most  experienced  or  most  confident  person 
for  the  role  did  not  always  play  the  commander.  In  the  military,  commanders  are 
respected  because  of  their  experience.  In  the  current  experiment,  this  element  was  not 
present,  and  respect  for  the  commander  may  have  been  compromised.  This  could  explain 
the  lack  of  attention  paid  to  the  commander  by  many  sub-unit  leaders.  If  the  commander 
had  more  knowledge,  it  may  also  have  been  less  intimidating  for  him  to  step  in  and  take 
charge.  Commanders  may  have  been  making  no  decisions  because: 

•  They  were  not  confident  in  their  own  ability;  and/or 

•  They  were  not  well  informed  of  the  situation  by  BCSS  updates  and  did  not  have 
enough  experience  to  make  situation  reports /updates  to  facilitate  situation  awareness. 

Another  observation  related  to  the  level  of  respect  subordinates  had  for  a  commander  was 
that  often  under  the  open  communication  condition,  commanders  would  be  bypassed  in 
the  communications.  Instead,  subordinates  would  communicate  directly  with  each  other. 
This  would  exclude  the  commander  from  the  decision-making  process,  and  break  down 
the  C2  structure. 

In  future  research,  a  sample  of  military  participants  should  be  in  roles  consistent  with  rank 
and  experience.  In  addition,  the  commander  should  be  provided  with  better  situation 
awareness  via  an  interactor  at  the  BCSS  monitor. 

5.9  Conclusion 

In  summary,  the  current  research  was  useful  as  means  of  determining  how  this  type  of 
experiment  could  be  continued.  It  also  revealed  trends,  and  identified  issues  that  need 
further  examination.  In  addition,  it  identified  the  following  methodological  details  that 
need  to  be  taken  into  account  in  future  research: 

•  Large  participant  numbers; 

•  Amended  interactor  training; 

•  Interactor  screening; 

•  Revised  training  regime; 

•  An  interactor  for  the  BCSS  monitor,  who  is  an  expert  BCSS  user; 

•  A  larger  team  structure  (to  see  more  complexity  emerge  in  the  communications.); 

•  Amended  maps  with  high  correlation  between  them;  and 

•  More  appropriate  scenarios  developed. 

This  study  also  provides  support  for  the  concept  that  Networked  Fire  Chief  is  an 
appropriate  tool  for  investigating  some  aspects  of  naturalistic  decision-making  theory. 
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However,  the  investigation  of  issues  related  to  expertise  in  a  military  context  may  not  be 
an  appropriate  area  for  investigation  with  this  microworld.  As  is  suggested  in  the 
literature,  this  study  indicates  that  expertise  is  largely  context  specific.  There  are 
significant  differences  between  a  fire  fighting  context  and  a  battle  context.  It  is 
recommended  that  future  work  should  further  examine  the  use  of  Janus  in  Human  Factors 
research.  Also,  the  validity  of  Networked  Fire  Chief  should  be  further  investigated  to 
determine  the  extent  to  which  results  can  be  generalised  between  areas  that  require 
naturalistic  decision-making. 
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Appendix  A:  Participants  Literature  and  Check-sheets 

A.l.  Information  for  Participants 

Thankyou  for  volunteering  to  participate  in  this  pilot  study.  Your  results  will  be  strictly 
confidential.  However  if  you  would  like  to  have  an  indication  of  the  study's  outcomes,  I 
can  e-mail  you  a  summary  of  results  around  the  end  of  February.  This  is  a  schedule  of 
what  you  will  be  tackling  over  the  three  days. 


Time 

Time 

Tuesday 

Time 

Wednesday 

9:30 

Training 

9:20 

Plan/ 

Restricted  (1) 

9:20 

Plan/ 

Open  (2) 

10:30 

Break 

11:30 

Lunch 

11:30 

Lunch 

11:00 

Plan/ 

Open  (1) 

12:30 

12:30 

Plan/ 

Open  (1) 

1:00 

Lunch 

2:35 

Break 

2:35 

Break 

2:00 

Plan/ 

Restricted  (2) 

2:50 

Plan/ 

Restricted  (1) 

2:50 

Plan/ 

Restricted  (2) 

4:00 

Finish/ 

Debrief 

5:00 

Finish/ 

Debrief 

5:00 

Finish/ 

Debrief 

(This  schedule  rotated  communication  architectures  for  groups) 


Further  Information 

The  current  study  uses  a  computer  war  game  simulation,  JANUS,  which  represents  two 
armies  warring  against  each  other.  A  comparison  will  be  made  between  experienced 
participants  (military)  and  inexperienced  participants  (civilian  DSTO  employees).  Under 
these  war  game  conditions,  participants  will  be  tested  under  the  two  different 
communication  architectures,  defined  in  Chapman's  experiment  (2000). 

For  each  communication  architecture  the  subjects  will  undergo  4  trials,  and  learning 
effects  will  be  monitored.  This  way  we  can  investigate  the  development  of  expertise,  and 
contrast  whether  this  development  is  different  for  military  personnel  compared  to  civilian 
subjects.  This  study  should  also  help  to  understand  whether  expert's  NDM  decision¬ 
making  performance  results  are  context  specific. 

You  will  be  acting  as  part  of  a  team  in  this  study.  Each  team  involved  in  the  study  consists 
of  3  members.  One  team  member  acts  as  the  commander,  while  the  other  two  act  as  sub¬ 
unit  leaders  controllers. 


The  plan  of  each  session  will  be: 


Intent/Ops 

Phase  1: 

Phase  2: 

orders  given  to 
team 

Team  planning 
carried  out 

Wargaming  / 
fighting  the  battle 
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Training 

Each  team  will  be  asked  to  take  part  in  a  training  session  to  familiarise  you  with  the 
specialised  set  up  of  JANUS.  This  training  session  will  follow  closely  a  training  protocol 
written  by  Chapman  (2000),  which  outlines  the  capabilities  of  Janus,  and  the  tactical 
information  you  will  need  to  understand. 

Intent  Statement 

The  intent  statement  for  the  exercise  will  be  given  to  the  participants  to  read. 

Planning 

In  the  next  phase,  you  will  be  given  a  lesson  in  the  basics  of  mission  planning.  You  will 
consequently  have  a  20  minute  planning  period,  in  which  you  will  work  with  your  fellow 
team  mates  to  form  a  plan.  During  the  planning  you  will  be  provided  with  a  large  map  of 
Kamaria  (the  land  in  which  the  battle  takes  place),  a  plastic  overlay,  and  some 
transparency  markers.  You  can  plan  routes  and  tactics  to  accomplish  your  mission  in  this 
session.  This  planning  phase  will  take  place  before  each  of  the  8  trials  begins. 

Janus  Trials 

Following  your  planning  you  will  be  required  to  perform  one  and  a  half-hour  war-gaming 
sessions.  This  length  of  time  is  needed  for  Janus  to  generate  some  meaningful  outcome 
data.  The  ordering  of  these  sessions  is  as  is  scheduled  above. 
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A.2.  Demographic  Information 

Participant  number: _ 

Allocated  Role  in  Janus  (please  cirdeJCommander  /Sub-unit  leader 

Rank  or  Field  of  Expertise: _ 

Sex  (please  circle)  Male  Female 
Age: _ years 

Please  read  the  following  questions  carefully  and  circle  the  appropriate  response. 

1.  Are  you,  or  have  you  ever  been  involved  in  the  military? 

Yes  /  No  (please  go  straight  to  Q.  4) 

2.  If  so,  what  was  your  main  role?  ( Please  describe.) 


3.  How  long  were  you  (or  have  you  been)  in  the  military? 


3.  How  often  are  you  involved  in  practical  combat  training? 

(Please  circle  appropriate  answer.) 

every  day  /  weekly  /  monthly  /  every  six  months  /  yearly 

4.  Are  you  or  have  you  ever  been  involved  in  another  organisation  which  requires  you  to 
make  emergency  decisions?  (For  example:  CFS,  hospital  emergency  room) 

Yes  /  No 

5.  Have  you  previously  used  JANUS? 

Yes  /  No 

6.  Have  you  worked  with  your  team  members  before? 

Yes  /  No 

7.  Have  you  ever  played  a  computer  game  or  simulation? 

Yes  /No. 

8.  When  was  the  last  time  you  played  a  computer  game  and/or  simulation?  Please  write 
number  in  appropriate  time  span. 

_ Days  ago  Weeks  ago _  Months  ago _  Years  ago _ 

9.  How  long  did  you  spend  on  the  game  or  simulation?  Please  tick  appropriate  box 

Less  than  30  minutes 
30  minutes  to  1  hour 
1  to  3  hours 
more  than  3  hours 

10.  When  was  the  time  before  that  that  you  played  a  computer  game?  ( Please  write  number 
in  appropriate  time  span.) 

_ Days  ago  W'eeks  ago _  Months  ago _  Years  ago _ 

11.  How  long  did  you  spend  on  the  game  or  simulation?  Please  tick  appropriate  box 

Less  than  30  minutes 
30  minutes  to  1  hour 
1  to  3  hours 
More  than  3  hours 

Questionnaire  is  completed.  Thank  you  for  your  effort 
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A.3.  Notes  to  the  Commander 

As  commander  you  are  responsible  for  overseeing  the  actions  of  your  subordinates,  and 
making  higher  level  decisions  based  upon  your  knowledge  of  the  "overall  picture".  You 
will  need  to  coordinate  your  ground  forces  (tanks  and  APC's)  with  your  support  forces 
(artillery  and  recon.  helicopters)  in  order  to  effectively  destroy  the  enemy. 

During  this  exercise  you  are  given  aids  to  help  you  create  a  good  situation  awareness  of 
the  battle  taking  place. 

You  will  have  a  BCSS  monitor  available.  This  is  a  restricted,  situation  awareness  tool  that 
provides  you  with  a  map  of  Kamaria  (This  correlates  with  the  Janus  screen).  The  system 
updates  the  positions  of  your  weapon  units  controlled  by  both  sub-unit  leaders  on  that 
map  periodically.  Enemy  sightings  do  not  appear  on  your  screen.  Rather,  you  will  need 
your  sub-unit  leaders  to  radio  through  the  coordinates  to  you,  and  you  can  update  them 
on  the  transparency  overlay  covering  the  BCSS  screen.  If  your  friendly  units  are  destroyed, 
they  will  cease  to  move  on  the  BCSS  screen.  Your  sub-unit  leaders  should  notify  you  of 
casualties  and  coordinates.  Then  you  can  identify  which  unit/s  have  been  destroyed,  and 
mark  them  on  the  overlay  if  desired. 

Below  the  BCSS  screen  you  will  have  a  picture  of  the  map  on  the  Janus  screen  your 
subordinates  are  using.  This  may  help  with  communication. 

You  also  have  a  whiteboard  if  you  wish  to  use  it.  You  may  want  to  write  tasks  here  during 
the  planning  phase  that  you  can  refer  to  them  later. 

You  are  provided  with  a  casualty  sheet  (if  you  wish  to  keep  a  track  of  how  many  units  are 
left),  and  a  sheet  outlining  weapon  ranges/sensor  ranges,  and  maximum  speeds.  These 
may  be  useful  also. 

It  is  recommended  that  you  keep  a  few  sheets  of  paper  near  the  BCSS  screen  so  that  you 
can  copy  down  coordinates  when  they  are  radioed  through,  and  update  them  on  the 
overlay  later. 

If  you  have  any  questions  about  any  part  of  this  ask  me,  and  we  will  try  and  sort  it  out. 
Rules  of  Engagement 

1.  You  must  not  move  the  vehicles  forward  of  the  beginning  positions  during  deployment. 
You  are  only  required  to  design  manoeuvre  routes. 

2.  The  use  of  mines  is  not  allowed.  (It  is  against  Kamarian  legislation) 

3.  You  should  give  coordinates  to  your  commander  regarding  enemy  sightings  and 
casualties  etc... 
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A.4.  Notes  to  the  Sub-unit  Leaders 

Your  role  is  to  make  lower  level  decisions,  and  to  report  all  relevant  information  (  eg. 
Coordinates  of  casualties,  Coordinates  of  enemy  sightings,  and  situation  reports)  back  to 
your  commander. 

It  is  very  important  that  you  do  communicate  this  information  to  him/her,  because  s /he  is 
responsible  for  overseeing  the  "big  picture",  and  s/he  needs  reports  in  order  to  do  so. 

Rules  of  Engagement 

1.  You  must  not  move  the  vehicles  forward  of  the  beginning  positions  during  deployment. 
You  are  only  required  to  design  manoeuvre  routes. 

2.  The  use  of  mines  is  not  allowed.  (It  is  against  Kamarian  legislation) 

3.  You  should  give  coordinates  to  your  commander  regarding  enemy  sightings  and 
casualties  etc. . . 


A.5.  Weapon  Unit  Guide 

Blue  Forces _ 

Combat  system  Combat  system  Weapon 

name _ type  range  (km) 


Sensor  range  Max  Speed 

(km)  (km/h) 


M1A1 

Abrahms 

FCV25 

FARH 
155  mm  Hzr 


Main  Battle  tank  3 

Infantry  fighting 

vehicle _ 

Armed  recon. 

Helicopter _ 

Howitzer  42 


**Time  to  plan  a  mission  before  firing  is  30  seconds. 

Time  of  flight  for  the  round  before  impact  can  be  5  to  58  seconds  (depending  on  range.) 


Possible  Red  forces _ 

Combat  system  Combat  system  Weapon 
name  type  ranee  (km) 


Main  battle  tank  4 


BMP-2 


Sensor  range  Max  Speed 


152  mmHzr 


Surface  to  air 
missile  vehicle 


Howitzer 
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A.6.  Casualties:  Blue  Force  Scenario  1  &  2 

You  may  keep  a  count  of  casualties  on  this  sheet  if  desired. 
Ground  Force:  Initial  Units  Description  _ 


M1A1 

Abrahms 

16 

Main  battle  tank 
(MBT) 

FCV25 

16 

ASLAV25  Light 
combat  vehicle 

Support  Force:  Initial  Units  Description 


.55  mm  HZR 

12 

155mm  Howitzer 

FARHI 

6 

Armed  Reconnaissance  Helicopter 
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A.7.  Radio  Protocol 

In  order  to  make  your  radio  transmissions  clearer,  we  ask  that  you  use  the  following 
format. 

1.  State  the  person's  call  sign  you  wish  to  talk  to. 

2.  State  your  own  call  sign. 

3.  Relay  your  message. 

4.  Finish  with  either  "over"  or  "out"  ("Over"  indicates  that  you  expect  a  response  from 
the  person  you  have  just  communicated  with.  "Out"  suggests  that  you  are  finished 
and  do  not  require  a  response.) 

An  example  of  a  communication  sequence: 

Hicon: 

"  Loconl,  loconl,  this  is  hicon.  Please  provide  me  with  a  situation  report  over." 

Locon  1: 

"  Receiving  you  hicon  please  standby  over" 

Locon  1: 

"We  are  currently  about  2  km  west  of  our  goal,  we  have  sighted  enemy  forces  to  the  south 
east,  and  are  currently  monitoring  them  with  our  FARH.  We  have  lost  16  units  in  total  so 
far,  over." 

Hicon: 

"Receiving  you  locon  1.  Continue  monitoring  the  enemy  to  your  south  east,  and  report 
immediately  once  you  have  identified  the  forces.  Hicon  out." 

Locon  1: 

"  Locon  1  out." 


Call  signs  you  will  need  to  be  aware  of: 

Locon  1: _ 

Locon  2: _ 

Hicon:  _ 
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Subject  Date  Time  Condition  Observer  _ 

Communication  Communication  Control  Morale  Situation  Awareness 

(problems)  (SA  maintenance)  (Pre-empting  behaviours) 
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Appendix  B:  Janus  Information 

B.l.  Training  Protocol 

During  this  study,  you  will  be  participating  in  a  two-sided  war  game.  Janus  is  an 
interactive  simulation  war  game  portraying  realistic  events  during  multi-sided  combat.  It 
uses  digitized  terrain  effecting  line  of  sight  and  movement,  depicting  contour  lines,  roads, 
rivers,  vegetation  and  urban  areas.  It  has  the  capability  to  be  networked  with  other 
systems,  in  order  to  simulate  a  war  game  with  multiple  sides. 

You  will  be  working  with  a  group  of  three  people  in  this  study,  representing  a  simplistic 
hierarchy.  One  team  member  will  act  as  the  commander,  the  other  two  as  his/her 
subordinate  unit  leaders.  Each  of  you  will  be  looking  at  your  view  of  the  battle  on  a 
computer  screen  (all  physically  isolated  from  one  another.)  Commands  will  be  issued  by 
the  commander,  and  feedback  is  required  from  the  sub-unit  leaders.  As  well  as  this,  in 
order  to  move  anything  on  the  screen,  carry  out  any  action,  or  request  computer 
information  simply  make  a  request  to  your  "interactor".  Your  interactor  links  you  to  the 
action  happening  on  the  computer.  This  person  is  responsible  for  translating  any  of  your 
commands  to  the  entities  on  the  Janus  screen.  They  will  act  on  commands  given  to  them, 
as  well  as,  suggesting  the  closest  possible  action  that  could  be  taken  using  Janus  if  requests 
you  make  are  impossible.  However,  they  cannot  guide  you  in  your  decision-making!! 
Your  role  is  that  of  the  battle  commander  for  the  blue  team.  In  this  role  you  are 
responsible  for  overseeing  the  moves  of  your  2  subordinate  sub-unit  leaders.  You  should 
keep  track  of  the  "overall  picture",  and  make  high-level  battlefield  decisions,  leaving  the 
more  intricate  decisions  to  your  subordinates.  Your  view  of  the  battle  will  be  provided  by 
BCSS.  This  is  a  computer  generated  map,  which  updates  the  positions  of  your  forces  and 
any  enemy  forces  that  are  sighted.  You  are  also  responsible  for  communicating 
information  of  enemy  sightings  to  both  of  your  subordinates  as  much  as  possible.  Your 
communication  will  take  place  using  a  radio  system. 

OR 

Your  role  is  that  of  the  sub-unit  leader  for  the  blue  team.  In  this  role  you  are  responsible 
for  receiving  and  acting  on  orders  given  to  you  by  the  commander.  You  should 
continually  give  feedback  on  your  actions  to  the  commander,  so  that  s/he  is  able  to  keep 
track  of  the  "overall  picture".  You  should  also  inform  your  commander  of  any  enemy 
sightings  as  much  as  possible.  Your  communication  will  take  place  using  a  radio  system. 
You  are  asked  to  treat  this  situation  as  realistically  as  possible,  as  we  are  looking  at  your 
situation  awareness  and  decision-making  processes.  Therefore  we  need  to  replicate  real 
world  cognitive  processes. 

Janus'  capabilities  are  outlined  below,  so  that  you  can  gain  an  understanding  of  what  you 
are  able  to  do  on  the  program.  Also  while  you  are  playing,  keep  the  Janus  info  sheet  close 
to  you  so  that  you  are  able  to  refer  to  it  if  in  doubt. 
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B.2.  Janus  Information 

Game  Time 

The  timer  at  the  bottom  right  hand  of  your  screen  expresses  time  in  simulation  clock:  days, 
hours  and  minutes. 

Communication 

You  will  be  tested  using  two  different  communication  architectures: 

Open  communication  -  where  all  team  members  hear  all  communications. 

Restricted  communication  -  where  communication  is  limited  between  you  and  one  of 
your  sub-unit  leaders  at  any  one  time.  Nobody  else  can  hear  that  transmission. 

Capabilities 

You  will  need  to  familiarise  yourself  with  the  capabilities  of  Janus  in  order  to  effectively 
plan  and  carry  out  your  mission.  The  capabilities  you  need  to  be  aware  of  are: 

•  Terrain  -Map  shows  topographical  features,  man  made  obstacles  and  natural 
obstacles.  Allows  for  different  zoom  levels. 

•  Movement  -  route  planning,  tracked,  wheeled,  foot,  aviation 

•  Line  of  Sight  -  Obscuration  by  smoke, ,  terrain  and  vegetation  are  taken  into  account 

•  Direct  fire  engagements  (automatic)  /  Indirect  fire  engagements  (manual). 

Terrain 
The  Map 

The  map  represents  a  fictional  land  called  Kamaria.  The  map  shows  topographical 
features,  man  made  obstacles  and  natural  obstacles.  You  will  notice  rivers,  roads,  areas  of 
vegetation  and  contour  lines.  You  should  take  all  of  this  into  account  when  planning  your 
movement  routes.  Your  "friendly"  forces  are  represented  as  blue.  Enemy  forces  that  have 
been  identified  are  red. 

Zoom 

Provides  you  with  the  capability  to  magnify  the  screen  in  order  to  see  more  clearly  what  is 
happening  on  the  map.  Your  magnification  level  is  located  above  the  time  on  the  bottom 
right  hand  side  of  the  screen. 
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Man  Made  Obstacles^  Everything  shown  in  black  appears  white  on  the  computer  screen 


Mine  1 


Doa 


Mine  2 


(active) 


(inactive) 


ooo 


■323? 


Natural  Obstacle 


Smoke  Pot 


1.  Mines 

The  enemy  may  have  mines  laid.  These  cannot  be  seen  on  the  screen  until  one  of  your 
units  discovers  them. 


2.  Clouds 

The  enemy  may  also  have  smoke  pots /grenades  laid  out.  These  will  be  seen  when  you 
activate  "clouds"  on  the  Janus  menu.  Dust  and  smoke  will  appear  on  the  screen  (in  the 
form  of  a  white  circle)  as  they  occur  during  game  time.  This  affects  sight  and  movement 
regardless  of  whether  you  have  the  "clouds"  menu  option  active. 

3.  Negotiating  Obstacles: 

Routes  through  man-made  barriers  will  only  be  visible  to  the  side  that  have  deployed 
them. 

Breaching:  Some  vehicles  however  have  the  ability  to  breach  obstacles.  When  they  reach  an 
obstacle  they  will  automatically  breach  the  obstacle  with  a  slight  delay. 

Movement 

Planning:  During  planning  you  will  need  to  consider  the  routes  you  intend  to  use  to 
accomplish  your  mission.  These  can  be  set  at  the  beginning  of  the  war  game,  and  may  be 
displayed  and  adjusted  as  the  game  continues.  Stop  nodes  can  be  included,  at  places 
where  you  would  like  the  vehicle  to  pause  until  instructed  to  proceed. 
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Manoeuvre;  You  will  continue  along  your  planned  route  unless  you  decide  to  stop,  or 
change  your  route  plan.  If  you  feel  it  is  necessary  to  show  your  movement  routes  this  is 
possible. 

Sprint;  Each  system  in  the  simulation  has  movement  speeds  and  capabilities  defined  in 
the  database.  The  speeds  depend  on  the  type  of  system  and  whether  it  is  moving  on  a  road 
or  cross-country.  A  group  speed  is  entered  at  the  server  to  set  units  in  a  Task  Force  to 
move  together  at  the  group  speedThis  keeps  task  forces  moving  together  in  formation.  It  is 
possible  to  have  whole  task  forces  "sprinting,  or  it  is  also  possible  to  sprint  one  of  your 
units. 

Mount/  Dismount:  You  are  able  to  mount  one  system  onto  another  (ie.  Soldiers  onto  troop 
carriers,  or  vehicles  onto  transports.)  When  you  are  doing  this  however  you  will  have  to 
take  into  consideration  the  weight  and  volume  of  the  objects  mounted  in  relation  to  the 
capacity  of  the  transporting  unit.  Once  you  have  reached  your  desired  destination  you  can 
order  a  dismount  of  either  a  single  entity,  or  all  those  loaded  onto  that  carrier. 

Status:  You  can  acquire  the  status  for  all  units  in  a  selected  task  force,  or  individual  units. 
The  information  includes:  unit  number,  system  name,  initial  strength,  remaining  strength, 
fuel  status,  velocity,  delay,  hold  fire,  suppression,  breach  and  chemical  status. 

Flight  Capabilities:  Aircraft  do  not  land  when  stopped  they  will  hover  or  circle  the  area. 
If  you  put  stop  nodes  behind  screening  crests  you  can  take  advantage  of  the  pop-up 
capability.  The  helicopter  will  remain  popped  up  for  60  seconds  and  will  observe  and  /  or 
fire  at  targets  that  he  acquires.  After  these  60  seconds  the  helicopter  will  automatically 
lower.  It  will  continue  to  pop  up  and  down  until  it  finds  a  target  and  shoots.  It  will  then 
proceed  to  its  next  stop  node. 

Line  of  Sight 

Line  of  Sight  &  Firing  Range:  Obscuration  by  smoke,  terrain  and  vegetation  are  taken 
into  account  when  a  vehicle's  line  of  sight  is  displayed.  Orange  lines  radiate  from  the  unit 
and  represent  what  the  unit  can  see  within  its  vision  limits.  This  is  bound  by  a  white  arc.  A 
break  in  the  line  is  dead  ground.  A  purple  dotted  arc  indicates  maximum  effective  range 
of  the  unit  s  primary  direct  fire  weapon.  This  can  be  shown  for  all  units  on  the  screen  in 
your  force.  (See  Figure  1.) 
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Figure  1:  Screen  view  for  line  of  sight 


An  element  s  view  fan  can  vary  from  360  degrees  to  a  very  narrow  arc  of  interest.  The 
lesser  the  arc  of  interest,  the  quicker  a  unit  will  be  able  to  detect  enemy  units  within  that 
arc  (Because  it  will  have  less  distance  to  scan.)  However  if  you  are  looking  in  a  direction 
different  to  where  the  enemy  is  approaching  you  will  not  be  able  to  see  them  and  are 


There  is  also  an  option  that  predicts  your  line  of  sight  at  any  point  on  the  screen.  The 

m  eractor  can  help  you  to  do  this  as  well  as  setting  you're  the  current  line  of  sight  and 
view  fan  for  each  element. 


Levels  of  Detection: 

As  your  units  move  across  the  terrain  they  will  detect  other  units  (enemy  or  friendly)  that 
are  m  their  line  of  sight.  These  will  appear  on  your  screen  as  follows: 

Yellow  box  -  Unknown  type. 

Yellow  icon  -  Appears  in  the  form  of  a  type  of  entity. 

Red  icon  -  Identified  as  enemy  icon. 

*  Be  careful  to  make  sure  not  to  shoot  at  "friendly"  units.  This  will  decrease  vour 
performance  scores!! 


Firing 

Rules  of  Engagement: 

Direct  fire  units  automatically  engage  enemy  units  once  they  have  been  acquired  and  are 
within  range. 

If  more  than  one  enemy  is  spotted  the  unit  will  engage  at  the  highest  priority  target.  If  a 
urnt  is  fired  at  it  will  also  automatically  engage  with  its  enemy,  (see  diagram  below) 
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Hold  Fire  (HF)  permits  you  to  order  the  unit  not  to  shoot  regardless  of  the  proximity  of  the 
enemy.  This  function  may  be  necessary  in  order  to  refrain  from  attacking  your  own 
friendly  forces.  It  is  important  that  you  communicate  with  your  team  members  before 
firing  at  an  unidentified  unit. 

Effects  of  an  Engagement;  As  a  result  of  a  firing  engagement  the  following  symbols  may 

appear  on  your  screen  at  the  point  of  that  battle: 


A  =  Killed  by  artillery 
C  =  Casualty 
S  =  Suppressed 
M  =  Minefield  casualty 
B  =  Chemical  kill 


Figure  2:  Screen  view  after  engagement 


Artillery  Information:  You  can  obtain  information  on  what  type  of  artillery  is  present  on 
each  of  the  "friendly"  entities. 

Kill  Statistics: 

A  running  total  of  your  unit  loses  is  kept.  You  can  obtain  3  different  reports  on  this: 

1)  The  initial  strength,  strength  remaining,  and  losses  for  the  highlighted  task  force. 

2)  The  same  information  but  for  the  group. 

3)  The  same  information  for  the  entire  side. 

Planning  Artillery/Indirect  Fire  Missions: 

You  can  plan/  alter  artillery  missions  of  the  type: 

•  HE  -  High  Explosive  (standard  artillery  round), 

•  IC  -  Improved  Conventional  HE  round  (more  accurate), 

•  RP  -  Rocket  Propelled  HE  (increased  range), 

•  CH  -  Chemical, 

•  PG  -  Precision  Guided  HE  by  laser  designation. 

Depending  on  the  artillery  unit  used,  the  type  of  fire  mission  will  determine  what  other 
functions  it  is  used  in  conjunction  with.  You  must  select  ammunition  type.  Number  of 
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volleys  and  aim  point.  When  a  firing  unit  has  twelve  missions  assigned,  its  mission  queue 
is  full  and  it  will  not  accept  any  additional  missions.  Great  discipline  must  be  employed 
to  prevent  excessive  use  of  Artillery  mission  planning  to  avoid  redundant  fire  missions 
from  being  fired.  You  have  a  limited  number  of  rounds  and  as  such  artillery  should  be 
employed  as  it  would  be  in  real  life. 

Planning. 

Prior  to  the  actual  planning  stage,  you'll  be  given  the  statement  of  intent  from  the  superior 
commander.  You  will  need  to  read,  understand  and  analyse  this  in  order  to  complete  the 
planning  and  wargaming  parts  of  the  experiment.  This  will  give  you  the  aims  and  goals  of 
the  mission  you  are  about  to  plan. 

Once  planning  begins,  you  should  find  (based  on  what  the  military  subjects  were  observed 
to  do  last  week)  that  there  are  4  main  phases  within  planning.  These  are  outlined  in  your 
handout.  3 

First,  there  is  the  review  and  confirm  stage.  This  essentially  involves  taking  the  intent 
statement  (and  the  aims  you  extracted  from  it)  and  relating  it  to  your  starting  positions, 
the  troops  you  have,  the  information  you  have  on  the  enemy  activities  and  positions,  and 
your  map.  You  should  at  this  stage  confirm  with  Doug  Williams  or  myself  (Monique) 
where  your  approximate  starting  locations  are,  and  where  you  will  be  heading  during  the 
game. 


Next,  you  should  be  evaluating  several  things:  the  possibilities  for  movement  of  your 
troops,  the  terrain  factors  that  will  impact  on  this,  the  potential  enemy  movements  and 
their  impact  on  what  you  want  to  do  (ie.  the  potential  risks  associated  with  your  troops 
doing  what  you  want  them  to),  the  speed  at  which  you  can  potentially  achieve  what  you 
want  to,  and  the  best  locations  for  the  deployment  of  your  troops. 

Third  you  will  need  to  develop  and  analyse  your  Courses  Of  Action.  Your  should  be  putting 
forward  ideas  about  where  and  how  fast  to  move  your  assets,  what  routes  they  should 
take,  potential  locations /routes  for  good  recon,  and  the  groupings  or  formations  you  want 
your  troops  in.  You  should  analyse  each  set  of  ideas  for  utility  and  the  possible 
consequences  and  decide  on  your  best  options.  Then  you  should  assign  tasks  to  your  troop 
groupings. 

Finally,  the  commander  will  need  to  confirm  with  his/her  subordinates  the  details  of  the 
missions.  That  is,  confirm  the  course  of  action  for  the  mission  (the  rough  routes  and 
headings  and  objectives,  etc),  the  taskings  of  each  group  (locon  1  and  2),  the  location  of 
potential  coordination  points  on  the  map  (if  you  think  these  are  necessary),  and  the  type  of 
communication  you'll  be  using  in  the  scenario  and  channels  your  radios  should  be  on. 

Following  this  planning  phase,  you  will  be  given  approximately  10  minutes  to  move  to 
your  assigned  rooms  (locon  1  or  2)  and  instruct  your  Janus  puckster  in  how  you  want  your 
troops  deployed.  After  this,  the  game  will  start  and  you  will  be  fighting  the  red  force. 
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Appendix  C:  Mission  Planning  Information  for 

Civilian  Participants 


C.l.  Prior  to  planning: 

Read  and  analyse  the  superior  commander's  intent  statement 
During  planning: 

1.  Review  and  confirm 

•  Initial  positions  for  each  group  (company/battalion/etc) 

•  General  headings  for  the  groups 

•  Own  forces:  ie.  numbers,  elements,  weapons,  functions  of  each  element,  etc 

•  What  is  known  about  the  enemy 

•  What  the  objectives  of  the  mission  are 

2.  Evaluate 

•  The  possibilities  for  movement 

•  Terrain  factors 

•  Potential  enemy  movements 

•  Ability  of  own  troops  to  achieve  the  objective 

•  Risks  associated  with  carrying  out  certain  actions 

•  To  what  locations  troops  should  be  deployed 

3.  Develop  &  Analyse  COAs 

•  Put  forward  ideas  on: 

•  Movement  routes 

•  Locations/routes  for  recon 

•  Troop/element  groupings  or  movement  formations 

•  Analyse  possible  outcomes/consequences  of  each 

•  Determine  best  cost/benefit  outcome  ratio  of  possible  COAs 

•  Assign  taskings  to  groups 

4.  Confirm 

•  Mission  plan  (movement  routes,  objectives,  ROEs,  etc) 

•  Taskings 

•  Location  of  coordination  points  on  map  (if  necessary) 

•  Type  of  communication  used  in  each  scenario  and  the  radio  channel  you  should  be  on 
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At  this  stage,  you  will  go  to  your  respective  rooms  with  your  puckster  to  deploy  your 
forces,  after  which  the  wargame  will  begin.  The  higher  commander  will  stay  in  the 
planning  room  (hicon). 

Post  planning: 

The  commander  (hicon)  should  write  up  taskings  and  situation  updates  on  the 
whiteboard  for  his/her  own  reference:  the  military  tended  to  use  the  following  format: 


H  Hour:  (start  time) 

Locon  1  (TF1)  name Locon  2  (TF2)  name 

^ . 

j  Your  locons 

[Ground  /  support  force]  [Ground  /  support  force] 
Tasks:Tasks: 

101  201 

202 

203 

204 

j  will  each  be 
;  one  of  these 

1  two  categories 

Contacts  &  Casualties 

Locon  lLocon  2 

Grid  referenceElement  &  numberGrid  referenceElement  &  number 

This  whiteboard  list  is  helpful  in  aiding  the  commander  to  keep  track  of  what's  going  on 
during  the  battle,  and  allows  him/her  to  make  decisions  and  alter  plans  as  necessary. 
Your  grid  references  and  elements  /  numbers  of  elements  column  should  allow  you  to 
track  casualties,  contacts  with  enemy,  and  own  troop  positions.  It  also  allows  you  to  track 
unidentified  entity  sightings  for  later  identification. 
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C.2.  Types  of  Actions  to  Consider  While  Planning  COAs 

Securing  flanks  of  certain  locations/ terrain  features  (roads,  towns,  etc) 

Prevent  enemy  capturing  towns/ features 
Provide  fire  support  to  other  task  force 
Securing  township(s) 

Recon  certain  areas  of  interest/ areas  of  potential  enemy  concentration  /  features 

Provide  security  to  flanks  of  force 

Attack  any  enemy  encountered 

Provide  support  to  attacks  planned  by  other  task  force 

Provide  indirect  fire  support 

Synchronise  attacks  (fire  missions) 

Coordinated  meeting  locations 

Frequency  of  reporting  between  locon  COs  and  hicon  CO  (especially  during  restricted 
communication  conditions) 

*  This  is  not  an  exhaustive  list,  but  should  provide  an  initial  guidance  for  you  to  work 
from. 


Brief  Glossary  of  Terms: 

COA:  Course  Of  Action  :  what  you  (and  your  troops)  will  be  attempting  to  do  to  achieve 
the  intent  (aims  of  the  mission) 

Intent:  The  aims  or  goals  of  the  mission  /  battle  you  are  about  to  undertake 
Locon:  Lower  control  (lower  level  company  commanders) 

Hicon:  Higher  control  (higher  commander’s  HQ) 

TF:  Task  Force 

ROEs:  Rules  of  Engagement  (eg.  shoot  the  enemy  as  you  identify  them) 

Ground  force:  MIAIs  (main  battle  tank)  and  FCV25s  (light  combat  vehicle) 

Support  force:  FARHIs  (armed  reconnaissance  helicopters)  and  155mm  Howitzer  artillery 
units 
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C.3.  General  Factors  to  Consider  While  Conducting  Mission  Planning 

Again,  this  is  not  an  exhaustive  list;  rather,  it  should  trigger  you  to  consider  the  important 
issues. 

Terrain: 

•  What  type  of  terrain  is  around  your  start  positions? 

Eg.  Are  there  areas  in  which  to  hide  /camouflage  your  vehicles? 

•  What  type  of  terrain  will  you  be  passing  through? 

Eg.  Crossing  rivers,  driving  through  forested  areas,  or  going  uphill  will 
slow  your  vehicles  considerably:  roads  provide  the  fastest  movement 
routes,  but  can  be  too  obvious 

•  Are  there  features  the  enemy  may  use  to  hide  behind? 

Eg.  These  features  can  also  hide  the  enemy's  vehicles/troops:  think  about 
conducting  recon  behind/ around  potential  masking  features 
Initial  enemy  information: 

•  What  has  the  enemy  done  up  to  now  (ie.  where  are  they  and  what  have  they  just  done 
to  get  there)  and  what  is  their  apparent  main  objective  in  the  near  future? 

•  What  enemy  assets  exist  and  where  are  they  located? 

Own  troops  information: 

•  How  many  of  each  element  do  you  have  at  your  disposal? 

•  What  are  their  start  positions? 

•  How  do  you  want  your  forces  grouped? 

•  What  are  your  elements'  capabilities? 

•  What  are  your  objectives  relative  to  the  terrain,  and  how  easily  can  you  get  to  them? 
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Appendix  D:  Statement  of  Intent  for  Janus  Study 

D.l.  First  Scenario 

You  have  deployed  in  the  mountains  East  of  the  township  of  Jalingo  to  mount  an  attack  to 
capture  the  town  from  the  Musorian  forces. 

Intelligence  sources  indicate  that  the  Musorian  army  has  a  Coy  +  Mechanised  battle  group 
stationed  around  the  town. 

Coordinate  your  ground,  air  and  indirect  fire  assets  to  locate,  engage  and  destroy  any 
enemy  elements  on  your  way  to  capturing  the  town  of  Jalingo. 

The  Musorian  Army  is  believed  to  have  some  indirect  fire  assets  located  in  the  town. 

D.2.  Second  Scenario 

Musorian  forces  have  increased  their  activities  on  the  Island  of  Kamaria  and  are  believed 
to  be  using  the  township  of  Tandaho  as  a  base  to  launch  an  offensive  on  the  main  city  of 
Lagowa.  Local  businessmen  travelling  between  Tandaho  and  Lagowa  have  reported 
seeing  a  number  of  Musorian  vehicles  just  off  the  main  highway  between  the  towns. 

Co-ordinate  your  combat  teams  to  conduct  a  search  and  destroy  mission  on  these 
Musorian  forces  before  they  can  establish  a  strangle  hold  on  the  major  route  between  the 
towns  and  launch  an  offensive. 
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Appendix  E:  Janus  Post  Trial  Questionnaire 

To  be  filled  out  after  finishing  experimental  trial  8. 

Participant  Number: 

Job  Title: 

Age: 

Sex: 

Date: 

Please  read  the  following  statements  carefully,  and  circle  the  most  appropriate  response. 


ENVIRONMENT  &  ARCHITECTURE 


1.  The  information  sheet  and  training  before  the  testing  sessions  made  me  aware  of  what 
my  role  was,  and  what  was  expected  of  me. 

Strongly  disagree  /  disagree  /  neutral  I  agree  I  strongly  agree 

2.  The  training  before  the  testing  sessions  made  me  aware  of  Janus'  capabilities. 

Strongly  disagree  / disagree  /  neutral  / agree  / strongly  agree 

3.  My  "interactor"  understood  and  followed  my  commands  well. 

Strongly  disagree  / disagree  /  neutral /agree  / strongly  agree 


Comments:. 


COMMUNICATION  &  SITUATION  AWARENESS 

1. 1  feel  that  I  was  aware  of  what  was  happening  on  the  computer  screen  in  the  Janus 
simulation. 

Strongly  disagree  / disagree  / neutral /agree  / strongly  agree 

2. 1  feel  that  I  could  communicate  well  with  my  fellow  team  members. 

Strongly  disagree  / disagree  /  neutral  / agree  /  strongly  agree 

3. 1  feel  that  my  fellow  team  members  responded  well  to  my  requests/suggestions. 

Strongly  disagree  / disagree  /  neutral  / agree  /  strongly  agree 

4. 1  feel  that  I  provided  good,  clear,  useful  information  to  my  fellow  team  members. 

Strongly  disagree  / disagree  /  neutral /agree  / strongly  agree 

5. 1  passed  information  to  the  appropriate  person  without  being  asked. 

Strongly  disagree  /  disagree  /  neutral  / agree  / strongly  agree 

6.  I  feel  that  open/  restricted  communication  enabled  me  to  achieve  a  better  situation 
awareness.  (Please  circle  the  appropriate  response.) 

8. 1  feel  that  open/  restricted  communication  enabled  me  to  perform  better  on  the  Janus 
war  game.  (Please  circle  the  appropriate  response.) 

Comments: _ 
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DEVELOPMENT  OF  EXPERTISE 

1. 1  feel  that  as  I  progressed  through  the  trials  my  understanding  of  the  war  game  situation 
increased. 

Strongly  disagree  /  disagree  /  neutral  / agree  /  strongly  agree 

2. 1  feel  that  as  I  progressed  through  the  trials  I  became  a  more  skilled  commander /sub¬ 
unit  leader  under  these  wargame  conditions. 

Strongly  disagree  /  disagree  /  neutral  I  agree  /  strongly  agree 

3. 1  feel  that  as  I  progressed  through  the  trials  I  began  to  feel  my  decision-making  became 
more  intuitive. 


Strongly  disagree  / disagree  /  neutral /agree /strongly  agree 

4. 1  feel  that  as  I  progressed  through  the  trials  I  began  to  filter  the  information  on  the 
screen  that  I  attended  to. 


Comments:. 


Strongly  disagree  /  disagree  /  neutral  /  agree  /  strongly  agree 


If  you  would  like  a  summary  of  results  from  this  study  when  they  are  compiled  please 
write  your  e-mail  address  below: 
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